Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,10 +52,10 @@ export MDC_API_KEY=your-api-key-here
MDC_API_KEY=your-api-key-here
```

3. **Get your dataset ID from the last section of the dataset URL at the MDC website**.
3. **Get your dataset ID and/or slug from the dataset's page at the MDC website**.

> [!TIP]
> You can find the `dataset-id` by looking at the URL of the dataset's page on MDC platform. The ID is the unique string of characters located at the very end of the URL, after the `/datasets/` path. For example, for URL `https://mozilladatacollective.com/datasets/cminc35no007no707hql26lzk` dataset id will be `cminc35no007no707hql26lzk`.
> You can find the dataset's **ID** and/or **slug** from its page on the MDC platform. Click the **Download** button and select **API / Python Access**, then switch to the **Python Library** tab on the pop-up modal. The **Dataset ID** and **Dataset Slug** appear at the bottom of the modal. Both can be used interchangeably in the Python library, but the slug is a more user-friendly identifier.

4. **Save a dataset locally**:
```
Expand Down Expand Up @@ -102,6 +102,10 @@ Returns a `Dataset`, or a `DatasetDict` keyed by split name for datasets with mu

## Programmatic submissions and uploads

> [!ΝΟΤΕ]
> In order to be able to upload datasets in the MDC platform you will first need to Request Access to Upload by
> navigating to your [profile](https://mozilladatacollective.com/profile/uploads) under the `Upload` tab.

You can create dataset submissions and upload files with resumable uploads into the MDC platform programmatically using our Python SDK:

```python
Expand Down
7 changes: 6 additions & 1 deletion docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,7 @@ export MDC_LOG_PATH=/path/to/save/the/log/files
**IMPORTANT NOTE:** Before trying to access any dataset, make sure you have thoroughly **read and agreed** to the specific dataset's conditions & licensing terms.

!!! tip
You can find the `dataset-id` by looking at the URL of the dataset's page on MDC platform. The ID is the unique string of characters located at the very end of the URL, after the `/datasets/` path. For example, for URL `https://mozilladatacollective.com/datasets/cmflnuzw6lrt9e6ui4kwcshvn` dataset id will be `cmflnuzw6lrt9e6ui4kwcshvn`.
You can find the dataset's **ID** and/or **slug** from its page on the MDC platform. Click the **Download** button and select **API / Python Access**, then switch to the **Python Library** tab on the pop-up modal. The **Dataset ID** and **Dataset Slug** appear at the bottom of the modal. Both can be used interchangeably in the Python library, but the slug is a more user-friendly identifier.

### Download a dataset

Expand All @@ -112,6 +112,11 @@ The files will be stored under `MDC_DOWNLOAD_PATH` (default `~/.mozdata/datasets
The SDK supports creating dataset submissions and uploading files with resumable uploads.
The upload state is stored in a JSON file alongside the archive so interrupted uploads can resume automatically.

> [!ΝΟΤΕ]
> In order to be able to upload datasets in the MDC platform you will first need to Request Access to Upload by
> navigating to your [profile](https://mozilladatacollective.com/profile/uploads) under the `Upload` tab.


```python
from datacollective import (
DatasetSubmission,
Expand Down
1 change: 1 addition & 0 deletions docs/upload.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ The SDK also supports **resumable uploads**, meaning if an upload is interrupted

Before uploading, ensure you have:

- Your Request Access to Upload **approved** in your MDC [profile](https://mozilladatacollective.com/profile/uploads).
- An API key from the Mozilla Data Collective [dashboard](https://mozilladatacollective.com/api-reference)
- Your dataset packaged as an archive file (`.tar.gz`, uploads use `application/gzip`)
- All the required metadata for the dataset submission
Expand Down
Loading