tap-sendgrid

This is a Singer tap that produces JSON-formatted data following the Singer spec.

This tap:

• Pulls raw data from the SendGrid v3 API
• Extracts the following resources:
  • Blocks
  • Bounces
  • Spam Reports
  • Invalid Emails
  • Global Suppressions
  • Lists
  • Segments
  • Single Sends
  • Single Send Stats
  • Stats Automations
  • Senders
  • Marketing Contacts Count
  • Marketing Field Definitions
  • Suppression Groups
  • Suppression Group Members
  • Templates
• Outputs the schema for each resource
• Incrementally pulls data based on the input state

---

Streams

blocks

• Primary keys: ['email']
• Replication strategy: INCREMENTAL (bookmark: created)
• Pagination: offset (limit / offset)

bounces

• Primary keys: ['email']
• Replication strategy: INCREMENTAL (bookmark: created)
• Pagination: offset (limit / offset)

spam_reports

• Primary keys: ['email']
• Replication strategy: INCREMENTAL (bookmark: created)
• Pagination: offset (limit / offset)

invalid_emails

• Primary keys: ['email']
• Replication strategy: INCREMENTAL (bookmark: created)
• Pagination: offset (limit / offset)

global_suppressions

• Primary keys: ['email']
• Replication strategy: INCREMENTAL (bookmark: created)
• Pagination: offset (limit / offset)

lists

• Primary keys: ['id']
• Replication strategy: FULL_TABLE
• Pagination: cursor (_metadata.next)

segments

• Primary keys: ['id']
• Replication strategy: FULL_TABLE
• Pagination: cursor (_metadata.next)

single_sends

• Primary keys: ['id']
• Replication strategy: FULL_TABLE
• Pagination: cursor (_metadata.next)

single_send_stats

• Primary keys: ['id']
• Replication strategy: FULL_TABLE
• Pagination: cursor (_metadata.next)

stats_automations

• Primary keys: ['id']
• Replication strategy: FULL_TABLE
• Pagination: cursor (_metadata.next)

senders

• Primary keys: ['id']
• Replication strategy: FULL_TABLE
• Pagination: none (single list response)

marketing_contacts_count

• Primary keys: none
• Replication strategy: FULL_TABLE
• Pagination: none (single aggregate response)

marketing_field_definitions

• Primary keys: ['id']
• Replication strategy: FULL_TABLE
• Pagination: none (merges reserved_fields and custom_fields arrays)

suppression_groups

• Primary keys: ['id']
• Replication strategy: FULL_TABLE
• Pagination: none (single list response)

suppression_group_members

• Primary keys: ['group_id', 'recipient_email']
• Replication strategy: FULL_TABLE
• Pagination: none (single list response per parent group)
• Parent stream: suppression_groups

templates

• Primary keys: ['id']
• Replication strategy: FULL_TABLE
• Pagination: cursor (_metadata.next)

---

Authentication

Generate an API key at Settings → API Keys in the SendGrid dashboard.

Required permission scopes:

Scope group	Streams
Suppressions → Read	blocks, bounces, spam_reports, invalid_emails, global_suppressions
ASM → Groups → Read	suppression_groups, suppression_group_members
Templates → Read	templates
Marketing → Read	lists, segments, single_sends, single_send_stats, stats_automations, senders, marketing_contacts_count, marketing_field_definitions

---

Quick Start

1. Install

   Clone this repository, then install using setup.py. A virtualenv is recommended:

   virtualenv -p python3 venv
   source venv/bin/activate
   python setup.py install

   Or with pip:

   pip install -e .

2. Dependent libraries

   The following libraries are installed automatically:

   pip install singer-python
   pip install target-stitch
   pip install target-json

   • singer-tools
   • target-stitch

3. Create your tap's config.json file

   The tap config file for this tap should include these entries:

   • api_key (string, required): SendGrid API key (starts with SG.)
   • start_date (string, required): RFC 3339 start date for incremental streams (e.g. 2024-01-01T00:00:00Z)
   • request_timeout (integer, optional): HTTP request timeout in seconds. Default is 300.
   • page_size (integer, optional): Records per page for incremental suppression streams (blocks, bounces, spam_reports, invalid_emails, global_suppressions). Default is 500. Full-table marketing streams use a fixed page size of 50 regardless of this setting (SendGrid caps those endpoints at 100).
   • lookback_window_days (integer, optional): Days to subtract from the bookmark on each incremental run to catch late-arriving records. Default is 0.

   {
     "api_key": "SG.xxxxxx",
     "start_date": "2024-01-01T00:00:00Z",
     "request_timeout": 300,
     "page_size": 500,
     "lookback_window_days": 1
   }

   Optionally, also create a state.json file. currently_syncing is an optional attribute used for identifying the last object to be synced in case the job is interrupted mid-stream. The next run would begin where the last job left off.

   {
     "currently_syncing": "blocks",
     "bookmarks": {
       "blocks":              { "created": "2024-01-01T00:00:00+00:00" },
       "bounces":             { "created": "2024-01-01T00:00:00+00:00" },
       "spam_reports":        { "created": "2024-01-01T00:00:00+00:00" },
       "invalid_emails":      { "created": "2024-01-01T00:00:00+00:00" },
       "global_suppressions": { "created": "2024-01-01T00:00:00+00:00" }
     }
   }

   The created bookmark stores an ISO 8601 datetime string (UTC). Incremental streams resume from the last saved bookmark value.

4. Run the Tap in Discovery Mode

   This creates a catalog.json for selecting objects/fields to integrate:

   tap-sendgrid --config config.json --discover > catalog.json

   See the Singer docs on discovery mode here.

5. Run the Tap in Sync Mode (with catalog) and write out to state file

   For Sync mode:

   tap-sendgrid --config config.json --catalog catalog.json > state.json
   tail -1 state.json > state.json.tmp && mv state.json.tmp state.json

   To load to JSON files to verify outputs:

   tap-sendgrid --config config.json --catalog catalog.json | target-json > state.json
   tail -1 state.json > state.json.tmp && mv state.json.tmp state.json

   To pseudo-load to Stitch Import API with dry run:

   tap-sendgrid --config config.json --catalog catalog.json \
     | target-stitch --config target_config.json --dry-run > state.json
   tail -1 state.json > state.json.tmp && mv state.json.tmp state.json

6. Test the Tap

   While developing the tap, the following utilities were run in accordance with Singer.io best practices:

   Pylint to improve code quality:

   pylint tap_sendgrid

   Pylint test resulted in the following score:

   Your code has been rated at 10.00/10
   

   To check the tap and verify working:

   tap-sendgrid --config config.json --catalog catalog.json | singer-check-tap > state.json
   tail -1 state.json > state.json.tmp && mv state.json.tmp state.json

   Unit Tests

   Unit tests may be run with the following:

   python -m pytest tests/unittests --verbose

   Note, you may need to install test dependencies:

   pip install -e '.[dev]'

---

Copyright © 2018 Stitch