Operations teams still receive supplier invoices, purchase orders, and shipping paperwork through shared mailboxes such as ap@ or orders@. When engineers wire that mailbox into generative AI extraction and ERP systems, the fragile layer is rarely the parser—it is the intake queue in front of it.
Building an email attachment intake webhook with the Gmail API fails when teams treat the webhook as a file-delivery pipe. Gmail does not push message bodies or PDFs; it pushes a minimal mailbox change notice through Google Cloud Pub/Sub that can be rate-limited, delayed, dropped, or redelivered.
Using third-party documentation and standards (not Insonl clients)—Google’s Gmail API push guide, client sync guide, REST references (users.watch, users.history.list, users.messages, users.messages.attachments), Cloud Pub/Sub push docs, and IETF RFC 5322—this post explains how to turn a shared Gmail mailbox into a durable attachment queue.
Treat Gmail push notifications as a wake-up signal, not a file pipe
The Gmail API push guide routes mailbox updates through a Cloud Pub/Sub topic (projects/<project-id>/topics/<topic-name>), where <project-id> must match the developer project calling users.watch. Grant publish rights on that topic to [email protected] (configuring an org-policy exception if domain-restricted sharing blocks it).
On mailbox changes, a wrapped Cloud Pub/Sub push request sends an HTTP POST whose message.data is a Base64URL-encoded JSON object containing emailAddress and historyId (example format from Google’s push guide):
{"emailAddress": "[email protected]", "historyId": "9876543210"}
Two rules follow immediately:
- The webhook carries no attachments and no Gmail message ID. Pub/Sub’s
message.messageIdis unrelated to Gmail messages; your backend still callsusers.history.listandusers.messages.get. - Never download files or run extraction inside the webhook handler. In Cloud Pub/Sub push subscriptions, only HTTP
102,200,201,202, and204acknowledge delivery; any other status is a negative acknowledgment, and per-message ack deadlines cannot be modified. Negative acknowledgments or expired ack deadlines can cause Pub/Sub to pause delivery across the subscription for100 msto60 susing global exponential push backoff, which cannot be turned off and uses a fixed delay calculation. Slow start adjusts concurrent push requests, reducing the push window when average push latency exceeds1 sor acks fall below99%.
Keep the endpoint thin: verify the signed JWT Cloud Pub/Sub sends in the Authorization header when push authentication is enabled, enqueue a (mailbox_id, notified_history_id) sync task, and return 200 or 204 immediately. Serialize sync execution per mailbox so overlapping wake-ups cannot race on the cursor.
Close the four gaps that cause missed events
A push-only listener can miss operational documents. Four documented behaviors in Gmail and Pub/Sub create gaps without explicit recovery:
| Failure mode | Documented behavior | Required safeguard |
|---|---|---|
| Watch expiry | users.watch stops unless renewed at least every 7 days (expiration is epoch ms). |
Renew daily (Google’s recommendation), store expiration, and alert before expiry. |
| Burst cap and loops | Gmail push caps each watched user at 1 event/s and drops excess notifications; a handler that changes a watched mailbox can trigger another notification and start a loop. |
Sync from stored startHistoryId, scope watch with labelIds and labelFilterBehavior: "INCLUDE" (not deprecated labelFilterAction), and track intake state in your own database rather than mutating the watched mailbox. |
| Delayed or dropped pushes | Gmail push notes Pub/Sub notifications may occasionally be delayed or dropped. | Run a scheduled fallback sweep calling users.history.list from the stored cursor on a fixed interval or after quiet periods. |
Expired historyId (HTTP 404) |
users.history.list records usually last at least a week, rarely only a few hours; an out-of-range startHistoryId typically returns HTTP 404. |
On HTTP 404, run a full sync via messages.list and batched messages.get over the period your intake must cover (a bounded lookback is a design choice defining recovery coverage), dedupe against stored Gmail IDs, and seed the cursor from the most recent message’s historyId. |
Two cursor rules prevent silent skips during partial syncs:
- Query from your committed cursor, not the webhook’s
historyId.users.history.listreturns records strictly afterstartHistoryId, and IDs increase chronologically with random gaps. Passing the notification’shistoryIdasstartHistoryIdexcludes records at or before that ID, including any unprocessed records after your last committed cursor. - Drain every page before advancing the cursor.
users.history.listdefaults to100records per page (max500). FollownextPageTokento completion, persist discovered message IDs first, and only then commit the response’s finalhistoryId.
Deduplicate at three layers: signal, MIME part, and document
Duplicate signals are normal: Pub/Sub resends unacknowledged pushes, calling users.watch immediately sends a notification, and in your own application a scheduled fallback sweep can overlap with a live push. Inside users.history.list, generic messages[] may duplicate entries in specific change arrays—so request historyTypes=messageAdded and read messagesAdded[] for new mail, and if applying an intake label also starts processing, request labelAdded and read labelsAdded[].
RFC 5322 rules explain why header checks alone cannot deduplicate attachments:
- Only
DateandFromare required (plusSenderon multi-addressFrom, §§3.6, 3.6.2).Message-IDisSHOULD, notMUST(§3.6.4), so intake code must handle missingMessage-IDheaders. - Headers vary in layout. Header fields have no guaranteed order (§3.6) and may fold across lines with
CRLFplus whitespace (§2.2.3); unfold before parsing and strip<and>fromMessage-ID(§3.6.4). - Resends and forwards differ.
Resent-*headers (§3.6.6) preserve the originalMessage-ID,From, andDate. A mail-client forward is an entirely new message from the forwarder (§3.6.6); if it has aMessage-ID, that identifies the forward, so compare attachment hashes to see whether the bytes were preserved. Dateis client time, not receipt time. RFC 5322 §3.6.1 definesDateas when the creator marked the message ready (including offline queues), not transport time. Gmail’sinternalDate(epoch ms) orders the inbox—reflecting when Google accepted normal SMTP-received mail, though API-migrated mail can derive it fromDate—making it a practical design choice for intake queue order.
Enforce three deduplication keys:
- Mailbox message key
(mailbox_id, gmail_message_id): Unique constraint absorbing Pub/Sub redeliveries, watch renewals, and full-sync replays. - Attachment key
(gmail_message_id, part_id)plus raw-byteSHA-256: Prevents double-processing one MIME part and flags byte-identical files across separate emails, while logging RFC 5322Message-ID,In-Reply-To, andReferencesfor audit lineage. - Business document key downstream: Two emails carrying re-exported PDFs of the same invoice have different Gmail IDs and can produce different
SHA-256hashes. Catch those after extraction with duplicate supplier invoice detection before ERP staging.
Walk the MIME tree and route shared-mailbox attachments safely
users.history.list typically populates only id and threadId on message stubs. Fetch unseen messages with users.messages.get using format=FULL (or format=RAW, a Base64URL-encoded full message to decode if you archive raw mail). Check scopes early: gmail.metadata works for watch and history.list, but users.messages.attachments.get requires gmail.readonly, gmail.modify, or https://mail.google.com/.
In users.messages, payload is a tree of MessagePart nodes. Container parts (multipart/*) hold child parts[] and may have an empty body; leaf parts hold immutable partId, mimeType, filename (present only if the part is an attachment), headers[], and body. Recurse through parts[] wherever present so nested attachments are never skipped.
Handle both MessagePartBody shapes:
- When
body.attachmentIdis present, checkbody.sizeagainst your size limit first, then fetch viausers.messages.attachments.getand Base64URL-decodedata. - When
body.attachmentIdis absent on a leaf attachment part, decode the inline Base64URL payload directly frombody.data.
When routing files from a shared mailbox into AI document and order processing, never route on To: or Cc: alone. Under RFC 5322 §3.6.3, a Bcc: recipient can receive a copy with the Bcc: header removed or left empty. Route in a clear order:
- Watched mailbox and intake label (sets the workflow lane, such as AP vs. orders).
- Sender headers (
FromandSender; under RFC 5322 §3.6.2,Reply-Tois the suggested reply address when supplied). - Payload inspection (validate file bytes against
mimeType,filename, andbody.size, and apply a separate policy for inline images). - Quarantine on bad files: Route encrypted PDFs, oversized archives, unsupported types, or downstream failures to a dead-letter queue for failed ERP imports with full message and
partIdcontext—never return HTTP500to Pub/Sub for a malformed supplier file.
Illustrative walkthrough: a shared ap@ mailbox during a morning burst
(Illustrative example—not a real company, mailbox, incident, or Insonl client. All addresses, IDs, counts, and timestamps are made up.)
Suppose [email protected] has a committed cursor at historyId = 50100 and a daily users.watch with labelIds: ["INBOX"] and labelFilterBehavior: "INCLUDE":
- Burst arrival (
09:00:01): Four supplier emails (m1–m4) arrive within600 ms. Gmail may drop excess notifications above1event/s; in this example, one notification wakes the endpoint after all four are visible in history. The endpoint verifies the Pub/Sub JWT, enqueues a sync job, and returns204in40 ms. - Cursor catch-up: The serialized worker calls
users.history.list(startHistoryId=50100, historyTypes=["messageAdded"], labelId="INBOX"), drains all pages, persistsm1–m4under(mailbox_id, gmail_message_id), and commits50140from the finalhistory.listresponse. - Duplicate signal (
09:00:09): Suppose Pub/Sub did not receive the204before the ack deadline and resends the push. The worker callshistory.list(startHistoryId=50140), finds no newmessagesAdded, and exits cleanly. - Bcc resend and quarantine: Suppose
m3arrived viaBcc:with theBcc:header removed (so[email protected]is absent fromTo:), carrying a PDF whoseSHA-256matches an attachment already ingested fromm1; intake linksm3as a duplicate delivery instead of starting a second extraction job. Messagem4holds an encrypted PDF; the worker saves raw bytes, marks(m4, partId=2)quarantined, and sends it to the exception queue whilem1andm2proceed to extraction.
Audit your email intake pipeline before scaling AI extraction
Insonl is a Ho Chi Minh City engineering studio building generative AI workflows and ERP systems. If operational attachments are getting missed in shared mailboxes—or creating duplicate ERP entries—start with our free AI workflow audit.
The audit is free; work after it is quoted separately.
Sources
- Google Workspace for Developers (third-party docs), Configure push notifications with the Gmail API: https://developers.google.com/workspace/gmail/api/guides/push
- Google Workspace for Developers (third-party docs), Synchronize clients with Gmail: https://developers.google.com/workspace/gmail/api/guides/sync
- Google Workspace for Developers (third-party docs), Gmail API Reference (
users.watch,users.history.list,users.messages,users.messages.get,users.messages.attachments,users.messages.attachments.get): https://developers.google.com/workspace/gmail/api/reference/rest/v1/users/watch · https://developers.google.com/workspace/gmail/api/reference/rest/v1/users.history/list · https://developers.google.com/workspace/gmail/api/reference/rest/v1/users.messages · https://developers.google.com/workspace/gmail/api/reference/rest/v1/users.messages/get · https://developers.google.com/workspace/gmail/api/reference/rest/v1/users.messages.attachments · https://developers.google.com/workspace/gmail/api/reference/rest/v1/users.messages.attachments/get - Google Cloud Documentation (third-party docs), Pub/Sub: Push subscriptions: https://docs.cloud.google.com/pubsub/docs/push
- IETF (third-party specification), RFC 5322: Internet Message Format: https://www.rfc-editor.org/rfc/rfc5322
- Insonl: Generative AI · ERP Systems · Free AI workflow audit · AI Document and Order Processing Operations · Dead-Letter Queue for Failed ERP Imports · Duplicate Supplier Invoice Detection
