--- title: "Product Transaction Delivery Through S3" slug: "product-transaction-delivery-through-s3" updated: 2025-04-25T19:12:57Z published: 2025-04-25T19:12:57Z canonical: "help.yahooinc.com/product-transaction-delivery-through-s3" --- > ## Documentation Index > Fetch the complete documentation index at: https://help.yahooinc.com/llms.txt > Use this file to discover all available pages before exploring further. # Product Transaction Delivery Through S3 For partners who do not support sending data via the Yahoo Conversion API (CAPI), Yahoo offers the ability to upload Transaction Data via S3. > [!WARNING] > Important > > This solution powers Yahoo In-Flight Outcomes (IFO) only and is not for standard conversion tracking. Set up the delivery of data files to designated AWS S3 locations for Yahoo to download. - [Architecture and Data Flow](/dsp-api/docs/product-transaction-delivery-through-s3#architecture-and-data-flow) - [Onboarding Requirements](/dsp-api/docs/product-transaction-delivery-through-s3#onboarding-requirements-from-data-provider) - [High-Level Guidelines: Privacy, Security and Performance](/dsp-api/docs/product-transaction-delivery-through-s3#highlevel-guidelines-privacy-security-and-performance) ## Architecture and Data Flow ![](https://cdn.us.document360.io/098e3080-9ea6-4eb4-acdb-13b988f387a2/Images/Documentation/s3_onboarding_dataflow.png?sv=2026-02-06&spr=https&st=2026-11-11T11%3A39%3A39Z&se=2026-11-11T12%3A39%3A39Z&sr=c&sp=r&sig=B6bmUTzlci09LZ8NbPGx91bPzw5q06DT3hZ9Tpkksh8%3D) ## Onboarding Requirements from Data Provider Please provide the following to allow for load estimation of the data you plan to send: 1. Upload frequency: assume daily. 2. Number of files per batch upload and their size after compression, .bz2 preferred. 3. Projected data volume per day. ## High-Level Guidelines: Privacy, Security and Performance The following are best practices for sharing data via S3: 1. Do not send data for opted out users. 2. Do not send duplicate user events. 1. If duplication cannot be avoided due to integrating with multiple data onboarding endpoints such as S3, Conversion API, and DOT Pixel, an extra string field ‘‘event_id”, needs to be sent for each event on all endpoints, so that Yahoo system can use it as a deduplication key. 3. Protect S3 bucket/location with security policy, such as 1. Disable public access. 2. Enforce HTTPS (TLS1.2 or above) connection. 3. Enable server side encryption (SSE-S3) with cypher key rotation at least every 12 months. 4. Group files by feed types and day/hour and upload them to folders named by feed types and date/time. Refer to [Directory Layout and File Format](/dsp-api/docs/product-transaction-delivery-through-s3#directory-layout-and-file-format). 5. Limit the file size to around 1 GB (after .bz2 compressed). 1. For small data sets, limit the number of files to under 5 per hour. 6. Avoid many small files. 7. Support credential rotation: annually or in need. 8. Secrets, e.g., credentials, should be delivered in encrypted format: GPG public key to be provided by the receiver of the credentials. Refer to [Sharing Secrets with External Partners using GPG](/dsp-api/docs/product-transaction-delivery-through-s3#sharing-secrets-with-external-partners-using-gpg). ### Directory Layout and File Format s3:///<3p-m>//yyyyMMdd _manifest -- 0 byte; upload completion marker; upload this after all other files are uploaded --- data file; TAB, ‘\t’, delimited ... #### File format and Contents of “_manifest” The unit of the file size is “byte”. <.schema size><.schema> --- delimited <.meta size><.meta> … #### File format and Contents of “<.schema>” The schema file, <.schema>, could be either of the following two formats. 1. A Pig header file, ”.pig_header”, in CSV column header format with extended data type support: 1. [,]{*} 2. Supported data types: [http://pig.apache.org/docs/latest/basic.html#data-types](http://pig.apache.org/docs/latest/basic.html#data-types) 3. [https://pig.apache.org/docs/r0.17.0/api/constant-values.html#org.apache.pig.data.DataType](https://pig.apache.org/docs/r0.17.0/api/constant-values.html#org.apache.pig.data.DataType) 2. The standard Apache Pig schema file, “.pig_schema”, in JSON file format. > [!NOTE] > Note > > The names, data types and order of the attributes in the “.schema” should match the columns in the data files. **Pig Data Types** | **Simple Types in .pig_header** | **Constant Value in .pig_schema** | **Description** | **Example** | | --- | --- | --- | --- | | [INT](https://pig.apache.org/docs/latest/api/org/apache/pig/data/DataType.html#INTEGER) | 10 | Signed 32-bit integer | 10 | | [LONG](https://pig.apache.org/docs/latest/api/org/apache/pig/data/DataType.html#LONG) | 15 | Signed 64-bit integer | Data: 10L or 10l Display: 10L | | [FLOAT](https://pig.apache.org/docs/latest/api/org/apache/pig/data/DataType.html#FLOAT) | 20 | 32-bit floating point | Data: 10.5F or 10.5f or 10.5e2f or 10.5E2F Display: 10.5F or 1050.0F | | [DOUBLE](https://pig.apache.org/docs/latest/api/org/apache/pig/data/DataType.html#DOUBLE) | 25 | 64-bit floating point | Data: 10.5 or 10.5e2 or 10.5E2 Display: 10.5 or 1050.0 | | [CHARARRAY](https://pig.apache.org/docs/latest/api/org/apache/pig/data/DataType.html#CHARARRAY) | 55 | Character array (string) in Unicode UTF-8 format | hello world | | [BYTEARRAY](https://pig.apache.org/docs/latest/api/org/apache/pig/data/DataType.html#BYTEARRAY) | 50 | Byte array (blob) | | | [BOOLEAN](https://pig.apache.org/docs/latest/api/org/apache/pig/data/DataType.html#BOOLEAN) | 5 | boolean | true/false (case insensitive) | | [DATETIME](https://pig.apache.org/docs/latest/api/org/apache/pig/data/DataType.html#DATETIME) | 30 | datetime | 1970-01-01T00:00:00.000+00:00 | | [BIGINTEGER](https://pig.apache.org/docs/latest/api/org/apache/pig/data/DataType.html#BIGINTEGER) | 65 | Java BigInteger | 200000000000 | | [BIGDECIMAL](https://pig.apache.org/docs/latest/api/org/apache/pig/data/DataType.html#BIGDECIMAL) | 70 | Java BigDecimal | 33.45678332 | ### Contacts of Operational alerts _s3_feed_alerts@.com ### Sharing Secrets with External Partners using GPG #### Background This document describes how company A, the secret sender, can share secrets with company B, the secret recipient, over the internet securely with GPG encryption. In a nutshell, 1. Company B uses GPG to generate a pair of public key, b_armor.pub, and private key, b_armor. 1. Company B needs to provide a password, password_B, in generating the public/private keys. 2. Company B will need this password in decrypting the encrypted secret from company A. 2. Company B emails b_armor.pub to company A 3. Company A uses b_armor.pub to encrypt the secret, and emails the encrypted blob to company B. 4. The company B uses b_armor (and password_B when prompted) to decrypt the encrypted blob. #### Detailed Steps 1. Company A requests company B to generate (if not yet) and export their GPG public key. Company B follows instruction here: [https://kb.iu.edu/d/awio](https://kb.iu.edu/d/awio) 1. Generate a key, assume company B’s email address is: “b_poc@b.com” 1. gpg --gen-key 1. Choose “(1) RSA and RSA (default)” 2. Key length: 2048 3. Expire in 2 weeks: 2w 4. Full Name 5. Email address: b_poc@b.com 6. Comment: GPG key w/ company A 7. Enter passphrase: (company B need in decrypting the secret) 2. gpg -o b_armor.pub -a --export b_poc@b.com 3. Company B emails “b_armor.pub” as attachment to company A. 2. Company A imports company B’s public key 1. gpg --import “b_armor.pub” 3. Company A puts secrets in “my_secret.txt” and encrypt it using b_armor.pub identified as “b_poc@b.com” 1. gpg -o my_secret_armor.txt -a -e -r b_poc@b.com my_secret.txt 4. Company A emails “my_secret_armor.txt” as attachment to company B