Overview

This article explains how to integrate your ProofPoint logs into Hunters. ProofPoint have various products and respectful data schemas - TAP, ProofPoint On Demand (PoD), ProofPoint GateWay (on premise). Following the guide bellow will allow Hunters to integrate your ProofPoint logs and ingest them to our database in a predefined schema, and then use these logs in our dedicated hunting mechanism.

Supported Data Types

A breakdown of supported products and data types by ProofPoint. Bellow find technical details on integrating the data.

  • ProofPoint On Demand - ProofPoint’s email cloud data services (see here for more details). Consists of raw email data, and is composed of 2 data types:

    • proofpoint-on-demand-message

    • proofpoint-on-demand-maillog

  • ProofPoint Targeted Attack Protection - ProofPoint’s email cloud protection services, contains alerts data and is composed of the following data types:

    • proofpoint-tap-messages-delivered

    • proofpoint-tap-messages-blocked

    • proofpoint-tap-clicks-blocked

    • proofpoint-tap-clicks-permitted

  • ProofPoint Email Gateway - ProofPoint on Premise server logs. Composed of 2 data types:

    • proofpoint-gateway-filter-logs

    • proofpoint-gateway-sendmail-logs

Integration Technical Details

ProofPoint On Demand (PoD)

The PoD API enables ingesting of 2 data types, Message and MailLog, which contain all raw email data that is gathered by ProofPoint. For more details on the data and the schema, see here.

Generating Credentials

In order to enable Hunters' collection and ingestion of PoD for your account, you will need to pass to Hunters the PoD Authentication keys - generated in the ProofPoint console - in a JSON format, which include the following keys:

{
  "clusterId": "<CLUSTER_ID>",
  "token": "<TOKEN>",
  "userId": "<USER_ID>"
}
CODE

The PoD token is designed to be uniquely used, and will be used permanently by Hunters for ingestion; hence this token cannot be used in any other platform or in manual API requests by any party.

ProofPoint Targeted Attack Protection (TAP)

The TAP API enables ingesting of 4 data types:

  • Messages Blocked

Messages Blocked Data Sample

{"completelyRewritten":false,"headerReplyTo":null,"spamScore":100,"malwareScore":0,"quarantineFolder":"Phish","subject":"NewFaxReceived-12/2/2021","headerFrom":"String","recipient":["String"],"fromAddress":["String"],"messageID":"String","cluster":"String","eventType":"messagesBlocked","threatsInfoMap":[{"threatTime":"2021-12-13T10:24:36.000Z","threatType":"url","threat":"URL","threatUrl":"URL","campaignID":null,"threatStatus":"active","classification":"phish","threatID":"String"}],"id":"String","messageSize":9569,"sender":"String","xmailer":null,"impostorScore":0.0,"messageTime":"2021-12-01T18:21:31.000Z","replyToAddress":[],"eventTime":"2021-12-13T10:40:10.744Z","quarantineRule":"inbound_phish","senderIP":"String","GUID":"String","messageParts":[{"md5":"String","filename":"String","sha256":"String","contentType":"text/html","oContentType":"text/html","disposition":"inline","sandboxStatus":"NOT_SUPPORTED"}],"toAddresses":["String"],"modulesRun":["av","spf","sandbox","dkimv","spam","urldefense"],"ccAddresses":[],"policyRoutes":["default_inbound"],"QID":"String","phishScore":100}

  • Messages Delivered

Messages Delivered Data Sample

{"spamScore":0,"phishScore":0,"threatsInfoMap":[{"threatID":"String","threatStatus":"active","classification":"phish","threatUrl":"URL","threatTime":"2021-12-12T22:31:18.000Z","threat":"URL","campaignID":null,"threatType":"url"}],"messageTime":"2021-12-12T22:16:58.000Z","impostorScore":0.0,"malwareScore":0,"cluster":"String","subject":"String","quarantineFolder":"Audit","quarantineRule":"audit","policyRoutes":["default_inbound"],"modulesRun":["av","zerohour","spf","spam","pdr","urldefense"],"messageSize":24546,"headerFrom":"String","headerReplyTo":null,"fromAddress":["String"],"ccAddresses":[],"replyToAddress":[],"toAddresses":["String"],"xmailer":null,"messageParts":[{"disposition":"inline","sha256":"String","md5":"String","filename":"String","sandboxStatus":null,"oContentType":"text/plain","contentType":"text/plain"}],"completelyRewritten":true,"id":"String","QID":"String","GUID":"String","sender":"String","recipient":["String"],"senderIP":"String","messageID":"String"}

  • Clicks Permitted

Clicks Permitted Data Sample

{"url":"String","classification":"spam","click_time":"2021-12-06T08:43:36+00:00","threat_time":"2021-12-08T19:46:14+00:00","user_agent":"String","campaign_id":"","id":"String","click_ip":"String","sender":"String","recipient":"String","sender_ip":"String","guid":"String","threat_id":"String","threat_url":"String","threat_status":"active","message_id":"String","domain":"String"}

  • Clicks Blocked

Clicks Blocked Data Sample

{"url": "String","classification": "phish","click_time": "2021-11-29T19:46:21+00:00","threat_time": "2021-11-27T00:53:00+00:00","user_agent": "String","campaign_id": "","id": "String","click_ip": "String","sender": "String","recipient": "String","sender_ip": "String","guid": "String","threat_id": "String","threat_url": "String","threat_status": "active","message_id": "String","domain": }

All of which contain all raw email threat data that is gathered by ProofPoint.

Generating Credentials

In order to enable Hunters' collection & ingestion of TAP for your account, you will need to pass to Hunters the TAP Principle and Secret. In order to generate the API credentials, follow the below steps:

  1. Log onto https://threatinsight.proofpoint.com

  2. Click on the Settings icon > Connected Applications

  3. Click Create New Credentials

  4. Choose a name and then click Generate

  5. Note the Principal and Secret.

Make sure that these are saved, as they are not available after the window is closed.

After generated, share the credentials with your Hunters Sales Engineer and Hunters Support Specialist.

ProofPoint Email Gateway

In order to enable Hunters' ingestion of the Email Gateway data for your account, you will need to set up a collector that will gather the data and ship it to an AWS S3 bucket. The data should be in a Text format, as exported by the product and as explained below.

The logs supported by Hunters contain two data types - filter_instance and sendmail data. The logs are stored as a text which contains key-value format for both data types, and with expected time format as: '%Y-%m-%dT%H:%M:%S.%f%z'.

An example for filter_instance logs:

Sample - proofpoint-gateway-filter-logs

2022-02-20T08:35:00.157994-06:00 myserver123 filter_instance1[21074]: rprt s=3ebnb6gs7y mod=session cmd=disconnect module= rule= action= helo=mydomain.com msgs=1 rcpts=1 routes=allow_relay,firewallsafe, duration=0.085 elapsed=0.176

An example for sendmail logs:

Sample - proofpoint-gateway-sendmail-logs

2022-02-20T08:34:59.967502-06:00 myserver123 sendmail[21646]: to=<mail@tempmail.com>, delay=1+16:51:01, xdelay=00:00:00, mailer=esmtp, tls_verify=NONE, tls_version=NONE, cipher=NONE, pri=3285075, relay=tempdomain.com. [165.231.167.17], dsn=4.0.0, stat=Deferred: Connection refused by temdomain.com.

Each data type has multiple combinations of keys that can appear in a specific record. The most important and prevalent keys are parsed by Hunters during the ingestion process.