Please access the dataset and code at the Zenodo link: https://zenodo.org/records/15594020. We will update it here shortly.

We have uploaded our dataset collection and analysis methods on Zenodo, including the privacy policy web crawler, the collected dataset of privacy policies, the LLM-based assessment framework, and the resulting LLM-based policy annotations as follows:

• /README.md provides a basic overview of the artifact's file structure, the same as the information provided below.
• /policy_crawlers: This directory includes the implementation of our distributed crawlers for privacy policy collection (in Python 3), as well as the top list we used (the CrUX list from August 2024). The Python code also includes the keywords we use for finding candidate privacy policies on the landing pages (as described in Section 5).
• /dataset: This directory includes all 5,210 unique privacy policy documents (from 4,896 distinct domains) that we evaluate in the results section, as described in the paper. We provide raw HTML or PDF (in /raw_html), converted Markdown format (in /markdown_format), and segmented content (in /segments) for each privacy policy document, as described in the paper. Each document is named as a hash value, which is generated by applying MD5 to the domain and the corresponding privacy policy HTML page or PDF URL. The file global_state.txt provides the crawling results for all domains in the top 100K of the CrUX list, as well as the Fortune US and EU domains. Users can find the corresponding domain and privacy policy URL hash in this file. Files in /segments_by_group contain the policy segment content organized by each domain group (top 1K, top 10K, top 100K, eu, us) for easy assessment of each domain group. These files can be used as the input for the Python code in /assessment_code_prompts.
• /assessment_code_prompts contains all the prompt texts we use for assessing the 34 clauses, including both the coverage task and the inspect tasks. We also provide the Python scripts for testing these prompts with our datasets, i.e., our assessment framework implementation.
• /LLM_experimental_annotations contains all the LLM annotations (i.e., inference results) produced by our experiments for all clauses, including both the coverage tasks and the inspection tasks for each privacy policy document. Note that for coverage tasks, each clause's directory contains 5,210 files, one for each policy document’s evaluation. For inspection tasks, since we first filter out those documents that do not include the specific clauses identified during the coverage tasks, there are fewer than 5,210 files per clause.