Data ingestion is the process of importing data from various internal and external sources into Salesforce Data Cloud. This ingested data is unified, processed, and made available for real-time personalization, segmentation, analytics, and AI-driven decisions. Salesforce Data Cloud provides standard connectors for data ingestion, like Sales/Service Cloud, Marketing Cloud, Amazon S3, etc. In some scenarios where connectors are not available, we can use the Data Ingestion API to import/migrate data into Data Cloud. This post will explore the usage of the data ingestion API in Salesforce Data Cloud.
Ingestion API is a RESTful interface to load data into Data Cloud. There are two types of Ingestion API in Salesforce
1. Streaming API
Streaming Ingestion API is used to ingest real-time data into Data Cloud. This will be an immediate response to the API call. This API can be used when there is a low volume of data to be ingested or where immediate response about ingestion is required.
2. Bulk API
Bulk Ingestion API is used when we need to upload a large volume of data to Data Cloud. This is an asynchronous API call rather than real real-time data update.
Use Case:
We want to upload customer transactions using the Streaming Ingestion API in Data Cloud Org.
Ingesting Data in Salesforce Data Cloud
We will use the Streaming Ingest API to handle the use case. Postman will be used as an external system that will consume the Ingest API. The following steps are required to ingest data:
- Set Up Ingestion API
- Create Schema for Ingestion API
- Create and Deploy Data Stream
- Create a Connected App in Salesforce
- Use Postman for Ingesting Streaming API
- Insert data using Ingest API
1. Set Up Ingestion API
We need to configure the Salesforce Data Cloud Org to accept and process data from external sources using Salesforce’s Data Ingestion API. This process involves establishing secure connections, defining data formats, and mapping incoming data to Salesforce’s Data Model Objects (DMOs) so that it can be used effectively for analytics, personalization, and customer insights.
We will create an ingestion API connector to connect and insert transaction records. We can create a connector using Setup -> Data Cloud -> External Integrations -> Ingestion API. You can also search for Ingestion API in the Setup Search box.
Create a new connection to the Ingest API source. Put the name ‘Customer Transaction‘.
It will create a connection, but the connection status for this will be schema required.
2. Create Schema for Ingestion API
We need to create an API schema for customer transactions. It will be created in a YAML file. For our use case, we need to create the following transactions.yaml file.
This YAML indicates that we have a transactions object with attributes(fields) – OrderId, CustomerId, TransactionAmount, PurchaseDate, and Location.
We need to upload this YAML file in the schema section of the above created ingestion connector. If file is valid, then it will upload and will show below screen
Once successfully uploaded, the connector status will change to Need Data Stream.
3. Create and Deploy Data Stream
Let us create a data stream from the Data Stream Tab. On the click of a new button, a new data stream dialog will open. Select Ingestion API from this screen.
The Ingestion API button will only show when we have at least one Ingestion API connector in the Org. It will show objects that are created using the schema upload. Select transaction object and proceed further.
The next screen will show the attributes of the object, select category, primary key, and event time field. Click next to the data space screen. Select a space for the data stream object and click deploy. It will create a data stream.
4. Create a Connected App in Salesforce
External systems require Salesforce authentication to use the Salesforce ingestion API. Create a connected app in Salesforce Data Cloud Org that will generate a consumer key and secret. This will require generating a bearer token to call the ingestion API. Check out steps 1-3 from the article Generate Salesforce Authentication Token using Postman.
Select the below OAuth scopes while creating a connected app.
- Access and manage your Data Cloud Ingestion API data (cdp_ingest_api).
- Access and manage your data (api).
- Perform requests on your behalf at any time (refresh_token, offline_access).
A Salesforce Administrator will share the authentication token URL, consumer key and secret, and user details with the external system stakeholder.
When you click on the Consumer Key and Secret button, it will verify you and then show the consumer key and secret.
5. Use Postman for Ingesting Streaming API
As mentioned, we will use Postman for ingesting the API as an external system. Search the Salesforce Data Cloud APIs collection in Postman. Fork the collection in your workspace and assign values in variables tab.
Navigate to the Authorization tab of collection and click Get New Access Token to generate a token. It will ask to log in to Data Cloud org. Once authenticated, it will generate access token.
Select the use token button to set the token variable.
6. Insert data using Ingest API
As we have generated an authentication token, let us consume the ingestion API to insert data into the data cloud object.
Create JSON for customer transactions and put that in the body tab of the Streaming API. Check the URL for the object that we have created. It can be similar to Customer_Transactions.Transaction. You can get this info from the Ingestion API connector.
This will respond with “accepted”: true. You can check new records in the Data Lake object.
Summary
Salesforce Data Cloud APIs give businesses powerful tools for handling data. It enables businesses to bring external customer data into Salesforce in a structured, secure, and scalable way. It is designed for systems that need to send batch or near real-time data (like customer records, transactions, and product info) into Data Cloud for processing and unification.
Need Help Integrating Your System?
Empower your business with a seamless Salesforce Data Cloud integration, backed by our proven expertise. Let’s connect and maximize your data’s value.
Related Posts
- How to Set Up Email Alerts for Salesforce Data Cloud Ingestion Failure
- Why Salesforce Data Cloud is the Future of Customer Engagement?
- Unlocking the Power of Salesforce Data Cloud: The Future of Customer 360
Recent Posts
- Ingest Data in Salesforce Data Cloud using Data Ingestion API
- How to Set Up Email Alerts for Salesforce Data Cloud Ingestion Failure
- Why Salesforce Data Cloud is the Future of Customer Engagement?
- Top 10 Features of Salesforce Sales Cloud Summer’ 25
- Unlocking the Power of Salesforce Data Cloud: The Future of Customer 360