CTA Notes : Integration Architecture 1

When you prepare for Certified Technical Architect (CTA) certification, you need to have a lot of details readily available in your memory.

Here we will go through Integration Architecture related topics you need to keep in mind for CTA review board. We will skip some beginner level topics in this series to reduce the length.

Integration Patterns

1) Remote Process Invocation—Request and Reply

Salesforce calls remote system, waits for it to process the request, and makes updates based on response.
Patterns

1. Enhanced External Services => Config way to invoke external REST services that gives OpenAPI 2.0 JSON schema format (Recommended Approach)
2. UI element => User Action => Invoke Apex => Call out (Recommended Approach)
3. Apex Trigger => asynchronous method => Call out
4. Apex Batch => Call out

Considerations

• Error Handling(Shown to user), Recovery (Changes are not saved in Salesforce till success response)
• Idempotent Design Considerations => Guarantee that repeated invocations are safe. (deduplication, pass unique Id etc)
• Security => One way SSL is enabled by default. Two way SSL (Self-signed/CA signed certs) can be enabled if needed, Can add one way hash/digital signature using crypto to ensure request authenticity, firewalls for remote system
• Governor Limit => Max timeout is 120 secs, Can have only 10 requests that takes more than 5 secs to execute (Long running callout). To avoid set timeout to 5 sec / use continuation

2) Remote Process Invocation—Fire and Forget

Salesforce calls remote system, remote system acknowledges the request, Salesforce continues processes without waiting.
Patterns

1. Platform events (Can be done through config/code)
2. Outbound Messaging => Sending SOAP message from DML operations that fire workflow (Create/Edit), Salesforce retries if the system didn't respond
3. Outbound Messaging with callback => Can send sessionId along with message. External system can use this session ID to access Salesforce REST/SOAP API to get more details about same record/other related records OR update result back to Salesforce.
4. UI Component triggering Async callout => controller => callout => External system acknowledges and continues processing asynchronously
5. Trigger => Async apex callout => external system
6. Batch job => Apexbatch => Callout to external system

Considerations

• Error Handling
  Callout => Remote system must handle errors that occur during asynchronous processing,
  Outbound message => Salesforce retries(15sec - 60min) up to 24 hours, external system must handle processing errors. Failures can be found in a queue for admins to monitor.
  Platform Events => Cannot rollback, replay ID can be used to get old events(upto 72 hours) after previous successful processing, Two options("Publish after Commit", "Publish Immediately")
• Idempotent Design Considerations => Guarantee that repeated invocations are safe. (deduplication, pass unique Id etc)
• Security Outbound Messaging=> One way SSL is enabled by default. Two way SSL can be enabled with "Outbound Messaging Certificate", IP whitelisting
• Governor Limit => Max timeout is 120 secs, Can have only 10 requests that takes more than 5 secs to execute (Long running callout). To avoid set timeout to 5 sec / use continuation

3) Batch Data Synchronization

Two way data synchronization between Salesforce and external system.
Patterns

1. Change Data Capture => Salesforce publishes change events for (CRUD, undelete) operations, near real time
2. Replication using ETL tool => To send to Salesforce, we can use Bulk API, SOAP API, See if we can find ETL that supports change data capture against external system To extract from Salesforce, Use time/status to filter rows. There is a getUpdated() method in SOAP API.
3. Remote Call-In & Remote Call-Out => Real time update, can cause a lot of taffice, special consideration on error handling, locking, can cause perofrmance issues

Considerations

• ETL tools uses control tables to store last job time, and other values related to previous runs that can be used to filter the data that needs to be synced.
• Use primary keys in both systems to match incoming data
• Surrogate keys are anonymous integer primary keys without business meaning to do transformation
• Get only updated data
• Group child table loads(contact) by parent(accountId) to avoid locking/failure
• Disable Triggers, workflow rules etc if not important

4) Remote Call-In

External system makes inbound request to make changes to Salesforce data. It is very common for companies to keep sales and service in Salesforce and order management in external system. In this case external system needs to update order status to Salesforce.
Patterns

1. SOAP API => Can publish event (Syntax is like object creation) Enterprise WSDL & Partner WSDL can be used. Can create custom SOAP endpoint for complex transaction management
2. REST API => Lightweight alternative for SOAP, No predefined contract, Composite resources option to make a series of updates in one API callback Composite resources => can use one request response for next input, couted as single API callback Bulk API is REST based
3. Apex Web Services => For complex transaction support/custom logic, But more maintenance
4. Apex REST services => Same as Apex Web Services but no need to consume service contract/WSDL
5. Bulk API => Large volume CRUD.
6. REST API to invoke an auto launched flow Salesforce

Considerations

• Authentication => SSL V3, TLS are supported in SOAP API, Ciphers must have at least 128 bit key length.
  REST API cal use OAuth to get a token or use session ID obtained from Outbound message/SOAP API.
  Queries through SOAP API, can timeout after 120 sec.
  Volume Limits => SOAP login request must be less than 10KB, Only upto 3600 login() calls/user/hour Max 2GB for contentversion, 1MB max for event message size, Bulk API (Upto 10k batches/24hours, upto 10k records in a batch)
  For SOAP API, remote system (client) must participate in a contract-first implementation where the contract is supplied by Salesforce (for example, Enterprise or Partner WSDL).
  Custom SOAP webservice/Apex REST service can support cross-object transactions and custom logic, But always check if middleware is suitable to do these.
  An integration user can have up to 10 query cursors open at a time.
  Only 250k events per hour in case of high volumne platform events. Can replay up to 72 hours
  Salesforce APIs do not support reliable messaging protocols like WS-ReliableMessaging

5) UI Update Based on Data Changes

Create custom interface using streaming API to notify user of the changes in data.
Patterns

1. Streaming API
2. Platform Events

Considerations

• Deliver and order of notifications is not guaranteed
• Notifications are not generated for bulk API operations

6) Data Virtualization

Showing external data in Salesforce in real time without persisting it.
Patterns

1. Salesforce Connect => maps data tables in external systems(SAP, MicroSoft etc) to external objects in Salesforce org
   External object features :- CRUD operations on external data, can define relations with custom objects, Can access in list views, tabs, page layouts, reports(limited) etc, Can view in Salesforce mobile app, available in global search and lookups, record feeds, apex, SOQL, SOSL, Salesforce APIs, change sets, packages etc, Can use Salesforce connect validator app to test connection.
   Can use one of the three connectors
   • OData 2.0/OData 4.0 connector to connect to OData producers
   • Cross-org adapter to connect to another Salesforce org using REST API
   • Custom adapter created via Apex => custom using Apex Connector Framework
2. Request and Reply => Callout to external system using SOAP/REST protocol to call out

Considerations

• Timeliness => User is waiting, max callout timeout is 120 sec, long running callout limits
• Relationships => Lookup(Using Salesforce record Id as parent), External Lookup(Using external ID standard field for lookup), Indirect lookup (custom external and unique field for lookup)
• Volume restrictions & rate limiting exist

Integration Pattern Categories

• Data Integration - Syncing data. Often includes often include aspects of master data management (MDM), data governance, mastering, de-duplication, data flow design, and others.
• Process Integration - Two or more apps integrated for completing business process.
  • orchestration (where one application is the central “controller”)
  • choreography (where applications are multi-participants and there is no central “controller”).
  complex to design & test, Needs to consider capabilities of systems, transactionality, Exception handling etc.
• Virtual Integration - Interacting with data in another system without copying to Salesforce.

Pattern Selection Matrix
Integration Terms

• Process choreography => mulitple systems working together without central authority
• Service orchestration => Single central system coordinating activities of other systems
• Transactionality => Support ACID (atomicity, consistency, isolation, durability). If complex multi-system transactions are required, go for middleware.
• Long polling/Comet Programming => Client keeps a connection open to server. Bayeux is a protocol for transporting asynchronous messages. CometD is a scalable HTTP-based event routing bus that implements Bayeux protocol. Streaming API uses this. Websockets => Establishes two way communication channel, Server Side Events => server publishes and clients listens are alternatives.
• Reverse Proxy Server => Receives requests on behalf of one or more servers, makes requests internally and returns results. Help to increase security, performance and reliability. Appache HTTP, nginx, IBM Webseal etc
• more on encryption