DualWrite – JohanPersson.nu

DualWrite Filtering

As I have mentioned before I am currently involved in implementing DualWrite with a customer and in this case there is data in CRM and FnO that we need to work around.

One example is a “single” contact that is used in multiple companies as an EDI adress. The reason I wrote “single” is that since we in FnO use the Global Adress Book, the contact is one Party that is instancuated as a contact in each of the Customers that are using it. This means that if you look in All Contacts there are a whole bunch of Duplicates.

These contacts are then synced, using DualWrite, to CRM… much to the annoyance of my CRM colleagues. This contact is of no use in CRM and I was asked to create a filter for them.

What i did was to set the field “Profession” in FnO (this was not being used) to EDI end then I added the following to the filters for Customer Contacts in DualWrite

FnO: (AssociatedContactType = 0) && (EMPLOYMENTPROFESSION != “EDI”)
CRM: msdyn_contactfor eq ‘Customer’ and msdyn_sellable eq false and msdyn_contactpersonid ne null and jobtitle ne ‘EDI’

As you notice, the syntax for the FnO and CRM sides is very different and that was the main reason I wrote this post. I needed a place to document the syntax 🙂

Links

Filter Not Working in FO side for Dual Write (dynamics.com)

Forcing Initial Sync in DualWrite

I am currently working on a project where we are implementing DualWrite in existing Dynamics 365 Environments. Since we do not have huge data volumes we decided to use Initial Sync in DualWrite to migrate some data from D365 Finance and Supply Chain over to D365 for Sales.

When we verified the data we noticed and issue with addresses. It turns out that the customers had all addresses set to purpose Business and as it turns out the addresses that are configured in the default mapping for DualWrite are Delivery and Invoice addresses. Once the customer fixed the addresses, I thought: “Lets just resync the Customer V3 entity using Initial Sync!”. Well… turns out is was not that easy.

When the sync har run it synced 3 customers instead of around 1800…

When I looked at the Data Management Project (after ALOT of troubleshooting) which Initial Sync generates I could see that the project was set to “Incremental Push Only”… I wanted it to do a complete sync… why is this happening?? What do I do know? There are 3 different ways that “might” solve this.

Disable and Enable Change Tracking

Your first option is to turn, change tracking off and then on again. You do this in the Data Management workspace, in Data Entities.
Reset DualWrite in FnO

In the DualWrite section of the Data Management workspace, click Reset Link. This will let you setup the link again; as well as purging all of the settings, it will also reset historical configuration. Since we already synced most of the data, we did not have to run initial sync for most of the entities… we only re-ran the Customers V3 entity
Deleting the DMF Project

Every initial sync created a DMF project and when run the Initial sync for the second time, Dataverse tried to be smart and reuse the DMF project which meant. Deleting the project meant that it had to be created again.
(Thanks Nathan Clouse for this insight)

Links:
https://learn.microsoft.com/dynamics365/fin-ops-core/dev-itpro/data-entities/dual-write/dual-write-troubleshooting-initial-sync?WT.mc_id=DX-MVP-5004702#error-customer-map

Handling internal Vendors in DualWrite

At the moment I am involved in a DualWrite implementation between FnO and CE. The goal is eventually to be able to generate Quotes from CE and have them sync to FnO. As you might know there are a lot of entities required to get to the point where we can sync Quotes and one of the is Vendors V2 and another is Released Procucts. In order to sync Release Products we first need to have Vendors.

At this customer they are buying a lot of their products from an internal vendor (aka another Legal Entity of the same FnO instance. When we first synced Vendors, everything worked perfectly, with 100% completion, (as far as we could see) but when we tried to synd Released Products we were missing Vendors

Quite a lot… So digging into this we found that these Vendor were never synced, which we found a bit strange (remember 100 % completion.

Turns out there is a filter in DualWrite that looks like this and apparently internal Vendors are not of the type Organizations… They are LegalEntity

So with some modifications… it looks like this

But there is another thing we need to fix… We need to add the following line in the tranform rule

Once that is done, I force another Initial Sync of Vendors and once that is completed I could successfully sync Products… Yay !!

Issues syncing Sales Order Lines with Dual Write

I am doing some experimenting with Synapse Links for Dataverse and to do that I need my FnO data in Dataverse. The way I do it, is using Dualwrite. When I try to sync Sales Order Lines V2 to salesorderdetails I get the following error:

Reason: Bad Request, Header x-ms-client-request-id f1004256-8a98-42fe-86a9-02b8e64a81a9, Produkten kan inte läggas till eftersom den inte är aktiv

(for those of you, not fluent in Swedish it says “The product cannot be added because it is not active)

This means that the product is synced but it is not active. To activate the product it needs to be published. To do this go through the following steps:

In Sales open products
Select a product and click Publish

The problem was I had a couple of thousand products. So I googled and found this forum thread helping me to write a workflow to automate it (this is way beyond my knowledge in CRM).

Links:
https://learn.microsoft.com/en-us/dynamics365/sales/publish-product-bundle-make-available-selling
Bulk publishing products – Dynamics 365 Sales Forum Community Forum

Data Events issues after refresh

In a previous article I wrote a bit about Data Events in Dynamics 365 for Finance and Operations. Data Events is a really simple way to create event-based integrations based on changes in the different data entities in FnO. The Data Events functionality is based on functionality in Power Platform. The setup requires the installation of the Finance and Operations Virtual Entity solution in the Power Platform environment connected to FnO. When you create a trigger Data Event it also created a virtual entity in DataVerse. This creates a couple of challenges when it comes to refreshing databases between environments.

For FnO
The settings for the endpoints created in FnO are partially stored in FnO and partially stored in an Azure Key Vault. The settings stored in FnO are amongst others Key Vault URL, App registration ID and Secret. To make sure that these settings do not get extracted from the environment (or accidentally moved to another environment) they are encrypted using an environment specific key and are thus not readable in the destination environment for the refresh. To restore the functionality of Data Events in the destination environment the endpoints need to be removed and recreated. After that has been done, we can re-activate the triggers.

Note that broken endpoints create an issue even if they are not being used. It seems like all endpoints are being validated when you try to create a new one which results in the creation failing.

For DataVerse/CRM
Since the functionality of Data Events is based on Virtual Entities created in DataVerse these will be overwritten when a refresh is done from one DataVerse environment to another. The error message you will get when the event is triggered is this:

Response status code does not indicate success: 404 ({“error”:{“code”:”0x80048d02″,”message”:”Virtual entity ‘mserp_vendvendorbankaccountentity’ not found for external entity VendVendorBankAccountEntity”}}).

(of course, with a different entity name based on your scenario)

The solution is to go to the Active Data Events tab in the Business Events workspace and remove and recreate each Data Event Trigger.

Note. You might have to wait a moment (1 minuter or so) before you recreate the trigger in order for everything to be properly cleaned in DataVerse.

Note: I have not been able to verify what happens if the identical triggers are set up in both source and destination environments. It might be that there are no issues or we might have the same issue. If anyone knows, please let me know 🙂

Lessons Learned: There has always been a lot to think about when you do refreshes… And DataVerse integration/DualWrite adds even more. There is a great article by Faisal Fareed here that detail steps that needs to be done for DualWrite integrated environments.

Links:
Not able to activate Data Events for entities – JohanPersson.nu
Microsoft Dynamics 365 and Power Platform Library: Steps to follow when refresh dual-write integrated environments (FO and CE) (daxture.blogspot.com)

DualWrite syncing empty lines on Initial Sync

I had this strange issue today…

One of my customers have set up DualWrite in their DEV environment and after some tweaking it worked OK. With that done they wanted to move the entire solution into TEST and verify that it worked. We packaged all the mappings into a Solution, exported it from DEV and moved it into TEST. We started enabling the Mappings for Legal Entity and a when we looked in CRM/CE we had a bunch of empty lines. We had the same number of lines but all of them were empty.

If we changed the data on one of the Legal Entities it synced over just fine but all the rest were still empty.

When we looked into the DMT files for the data project for the initial sync the files looked “good” to me… they contained data

|NAME|,|LEGALENTITYID|
|LegalEntity1|,|AAA|
|LegalEntity2|,|AAB|
|LegalEntity3|,|AAC|
|LegalEntity4|,|AAE|
|LegalEntity5|,|AAF|
|LegalEntity6|,|AAG|
|LegalEntity7|,|AAH|
|LegalEntity8|,|AAI|
|LegalEntity9|,|AAJ|
|LegalEntity10|,|AAK|
|LegalEntity11|,|AAL|
|Company accounts data|,|dat|

Some of you might already see the issue 🙂 (don’t spoil the surprise) When contacting Support they told me then there is a problem with the text qualifier in the file… the strings should be enclosed in ” instead of |

It turns out that someone had changed the default format CSV-Unicode in Data Management Framework to this:

I changed it back to this:

After cleaning out the records from CRM/CE and rerunning initial sync everything works again…

Warning:[DWCE0001] Export was skipped. Max lookup count supported in Initial Sync stage is 10. Current lookup count 11

Today I was setting up the Dual-Write sync for one of my customers and I bumped into this error message:

Warning:[DWCE0001] Export was skipped. Max lookup count supported in Initial Sync stage is 10. Current lookup count 11

The issue here is that there is a limit in DataVerse that (I would guess) for performance issues there is a hard limit on a maximum of lookup fields.

In our case we had these fields in the default mapping for Customer V3 -> Account using lookup:

transactioncurrencyid.isocurrencycode
msdyn_customergroupid.msdyn_groupid
msdyn_billingaccount.accountnumber
msdyn_paymentday.msdyn_name
msdyn_customerpaymentmethod.msdyn_name
msdyn_paymentschedule.msdyn_name
msdyn_paymentterm.msdyn_name
msdyn_vendor.msdyn_vendoraccountnumber
primarycontactid.msdyn_contactpersonid
msdyn_salestaxgroup.msdyn_name
msdyn_company.cdm_companycode

The workaround is to remove one of the problematic mappings, do the initial sync and add it back. Remember to take a screenshot of the mapping that you are removing so you can put it back exactly the same.