DualWrite syncing empty lines on Initial Sync

I had this strange issue today…

One of my customers have set up DualWrite in their DEV environment and after some tweaking it worked OK. With that done they wanted to move the entire solution into TEST and verify that it worked. We packaged all the mappings into a Solution, exported it from DEV and moved it into TEST. We started enabling the Mappings for Legal Entity and a when we looked in CRM/CE we had a bunch of empty lines. We had the same number of lines but all of them were empty.

If we changed the data on one of the Legal Entities it synced over just fine but all the rest were still empty.

When we looked into the DMT files for the data project for the initial sync the files looked “good” to me… they contained data

|NAME|,|LEGALENTITYID|
|LegalEntity1|,|AAA|
|LegalEntity2|,|AAB|
|LegalEntity3|,|AAC|
|LegalEntity4|,|AAE|
|LegalEntity5|,|AAF|
|LegalEntity6|,|AAG|
|LegalEntity7|,|AAH|
|LegalEntity8|,|AAI|
|LegalEntity9|,|AAJ|
|LegalEntity10|,|AAK|
|LegalEntity11|,|AAL|
|Company accounts data|,|dat|

Some of you might already see the issue 🙂 (don’t spoil the surprise) When contacting Support they told me then there is a problem with the text qualifier in the file… the strings should be enclosed in ” instead of |

It turns out that someone had changed the default format CSV-Unicode in Data Management Framework to this:

I changed it back to this:

After cleaning out the records from CRM/CE and rerunning initial sync everything works again…

Warning:[DWCE0001] Export was skipped. Max lookup count supported in Initial Sync stage is 10. Current lookup count 11

Today I was setting up the Dual-Write sync for one of my customers and I bumped into this error message:

Warning:[DWCE0001] Export was skipped. Max lookup count supported in Initial Sync stage is 10. Current lookup count 11

The issue here is that there is a limit in DataVerse that (I would guess) for performance issues there is a hard limit on a maximum of lookup fields.

In our case we had these fields in the default mapping for Customer V3 -> Account using lookup:

transactioncurrencyid.isocurrencycode
msdyn_customergroupid.msdyn_groupid
msdyn_billingaccount.accountnumber
msdyn_paymentday.msdyn_name
msdyn_customerpaymentmethod.msdyn_name
msdyn_paymentschedule.msdyn_name
msdyn_paymentterm.msdyn_name
msdyn_vendor.msdyn_vendoraccountnumber
primarycontactid.msdyn_contactpersonid
msdyn_salestaxgroup.msdyn_name
msdyn_company.cdm_companycode

The workaround is to remove one of the problematic mappings, do the initial sync and add it back. Remember to take a screenshot of the mapping that you are removing so you can put it back exactly the same.