Documentation Content

#Introduction

This page covers all topics of how the module transforms and upserts documents to Shopware.

Basically a document goes through several steps and subsections before it is eventually sent to Shopware:

• A document is validated in the subsection -validate
• A document is upserted together with other documents via Sync API in the subsection -upsert
• A document is upserted individually in the subsection sync-api-retry-operations if something went wrong in -upsert

This section is intended to explain how the upsert and sync-api-retry-operations subsections are working and how the module uses the Sync API of Shopware.

#The Upsert Subsection

The upsert subsection basically does three things with each source object to generate a Shopware entity:

• Take the source object and convert it into an Entity. Basically the module uses the models of the Shopware core as data transfer objects whose super class is Entity.
• Take the generated Entity and transform it into an array.
• Generate a Sync API request with all transformed entity-arrays (of the same type) and send it to Shopware

#Sync API Upserts

As already mentioned the module collects entities of the same type and upserts them as a bulk payload via Sync API. The Sync API requests are sent with the indexing-behavior header use-queue-indexing.

#Sync API Errors and Retries

To prevent single documents from causing the content of a complete Sync API request to get lost, a retry mechanism is available since module version 2.1.0.

The validation does a pretty solid job of sorting out invalid documents that would cause a request to Shopware to fail. However, errors can never be completely eliminated. Especially new data sets from new projects tend to create new combinations of data that are not yet included in the validation. Another cause for failing entities are faulty Shopware plugins and configurations.

#Sync API Retry Subsection

The retry mechanism takes place in the subsection sync-api-retry-operations. Basically this subsection sends each entity from a failed request individually to Shopware again.

If the upsert of a document finally fails in a retry you will be notified. Documents that failed to get upserted in a retry are referred to as "corrupted entities".

#Corrupted Documents

A corrupted entity / document is an entity that finally failed to get upserted in a retry. A corrupted entity could not be created in Shopware and caused its request to fail. This could either be caused by a missing validation case or some kind of misconfiguration. Plugins in Shopware and configurations are another cause for failing requests. Corrupted entities are marked as invalid. A validation info is added to the affected source document.

#Corrupted Entity Event

Every time a corrupted entity (= entity, that could not be sent to shopware in a retry individually) an event is dispatched: Synqup\Modules\Shopware6Bundle\Output\Core\SyncApiProcessor\CorruptedEntityDetectedEvent. You can use this event if you want to implement an individual subscriber to handle corrupt documents.

#Retry Progress Handling

Note that in order to avoid this subsection fo finish too early it is initialized with a default value of 1. This means that the progress is always too high by the value 1. As soon as the module has finished all of its subsections the progress is reduced by 1.

The progress of sync-api-retry-operations is not equal to the amount of failed requests. It is actually equal to the amount of entities that will be sent to Shopware individually. Example: If a Sync-Request with 100 entities fails due to one broken document the retry-count is 100, although only one request failed.

The subsection sync-api-retry-operations is shared between all subsections. So if a request with 10 products and another with 5 manufacturers fails the progress count will be 15.

#How Retries Work

The following section explains how the retry mechanism works. This will help you to understand generated logging-messages.

Step 1 - Sync API Error

If a request to the Sync API to upsert N entities fails you will see the following warning message: "The entity operations could not be performed in a single request. Every entity will be resent individually."

Step 2 - Increase Progress

The failed request will cause the subsection sync-api-retry-operations to get increased by N.

Step 3 - Retry each Document

After the initial Sync API request failed, each document is sent to Shopware individually. If the module was able to upsert the document successfully it will log the following message: "successfully retried to [upsert|delete] [documentType] (ObjectId: %s)"

If the individual retry-request fails a "corrupted" document was found. In this case you see the following error-message: "Corrupted entity detected! Upsert of entity {{source-object-info}} finally failed in a retry: {{EXCEPTION_DETAILS}}"

Each retry will increase the progress of the subsection sync-api-retry-operations by 1, regardless if it succeeded or failed.