Microsoft Dynamics CRM 2011 is the latest version of Microsoft’s CRM platform. The SaaS version is already live and the on-site version will likely be released within a couple weeks. Unlike previous versions of Dynamics CRM, the 2011 release does NOT have a BizTalk-specific send adapter. The stated guidance is to use the existing SOAP endpoints through the BizTalk WCF adapter. So what is this experience like? In a word, mixed. In this post, I’ll show you what it takes to perform both “query” and “create” operations against Dynamics CRM 2011 using BizTalk Server.

Before I start, I’ll say that I really like using Dynamics CRM 2011. It’s a marked improvement over the previous version (CRM 4) and is a very simple to use application platform. I’m the architect of a project that is leveraging it and am a fan overall. It competes directly with Salesforce.com, which I also like very much, and has areas where it is better and areas where it is worse. I’ll say up front that I think the integration between Salesforce.com and BizTalk is MUCH cleaner than the integration between Dynamics CRM 2011 and BizTalk, but see if you agree with me after this post.

Integration Strategies

Right up front, you have a choice to make. Now, I’m working against a Release Candidate, so there’s a chance that things change by the formal release but I doubt it. Dynamics CRM 2011 has a diverse set of integration options (see MSDN page on Web Service integration here). They have a very nice REST interface for interacting with standard and custom entities in the system. BizTalk Server can’t talk “REST”, so that’s out. They have (I think it’s still in the RC) as ASMX endpoint for legacy clients, and that is available for BizTalk consumers. The final option is their new WCF SOAP endpoint. Microsoft made a distinct choice to build an untyped interface into their SOAP service. That is, the operations like Create or Update take in a generic Entity object. An Entity has a name and a property bag of name/value pairs that hold the record’s columns and values. If you are a building a .NET client to call Dynamics CRM 2011, you can use the rich SDK provided and generate some early bound classes which can be passed to a special proxy class (OrganizationServiceProxy) which hides the underlying translation between typed objects and the Entity object. There’s a special WCF behavior (ProxyTypesBehavior) in play there too. So for .NET WCF clients, you don’t know you’re dealing with an untyped SOAP interface. For non-.NET clients, or software that can’t leverage their SDK service proxy, you have to use the untyped interface directly.

So in real life, your choice as a BizTalk developer will have to be either (a) deal with messiness of creating and consuming untyped messages, or (b) build proxy services for BizTalk to invoke that take in typed objects and communicate to Dynamics CRM. Ideally the Microsoft team would ship a WCF behavior that I could add to the BizTalk adapter that would do this typed-to-untyped translation both inbound and outbound, but I haven’t heard any mention of anything like that.

In this post, I’ll show option A which includes dealing directly with the bare Entity message type. I’m scared. Hold me.

Referencing the Service

First off, we need to add a reference to the SOAP endpoint. Within Dynamics CRM, all the links to service endpoints can be found in the Customization menu under Developer Resources. I’ve chosen the Organization Service which has a WSDL to point to.

Within a BizTalk project in Visual Studio.NET, I added a generated item, and chose to consume a WCF service. After added the reference, I get a ton of generated artifacts.

Now in an ideal world, these schemas would be considered valid. Alas, that is not the case. When opening the schemas, I got all sorts of “end of the world” errors claiming that types couldn’t be found. Apparently there is a lot of cross-schema-referencing missing from the schemas. Wonderful. So, I had to manually add a bunch of import statements to each schema. To save someone else the pain, I’ll list out what I did:

To OrganizationService_schemas_datacontract_org_2004_07_System_Collections_
Generic.xsd schema, I added an Import directive to OrganizationService_schemas_microsoft_com_xrm_2011_Contracts.xsd.

To OrganizationService_schemas_microsoft_com_2003_10_Serialization_Arrays.xsd schema I added an Import directive to OrganizationService_schemas_microsoft_com_2003_10_Serialization.xsd.

To OrganizationService_schemas_microsoft_com_crm_2011_Contracts.xsd schema I added Import directives to both OrganizationService_schemas_microsoft_com_2003_10_Serialization_Arrays.xsd and OrganizationService_schemas_microsoft_com_xrm_2011_Contracts.xsd.

To OrganizationService_schemas_microsoft_com_xrm_2011_Contracts.xsd schema, I added an Import directive to OrganizationService_schemas_microsoft_com_2003_10_Serialization_Arrays.xsd, OrganizationService_schemas_microsoft_com_xrm_2011_Metadata.xsd and OrganizationService_schemas_datacontract_org_2004_07_System_Collections_
Generic.xsd.

To OrganizationService_schemas_microsoft_com_xrm_2011_Contracts_Services.xsd schema I added Import directives to both OrganizationService_schemas_microsoft_com_2003_10_Serialization_Arrays.xsd and OrganizationService_schemas_microsoft_com_xrm_2011_Contracts.xsd.

To OrganizationService_schemas_microsoft_com_xrm_2011_Metadata.xsd schema I added an Import directive to OrganizationService_schemas_datacontract_org_2004_07_System_Collections_
Generic.xsd and OrganizationService_schemas_microsoft_com_xrm_2011_Contracts.xsd.

Ugh. Note that even consuming their SOAP service from a custom .NET app required me to add some KnownType directives to the generated classes in order to make the service call work. So, there is some work to do on interface definitions before the final launch of the product.

UPDATE (2/17/11): The latest CRM SDK version 5.0.1 includes compliant BizTalk Server schemas that can replace the ones added by the service reference.

For my simple demo scenario, I have a single message that holds details used for both querying and creating CRM records. It holds the GUID identifier for a record in its Query node and in its Create node, it has a series of record attributes to apply to a new record.

Mapping the Query Message

Retrieving a record is pretty simple. In this case, all you need to populate is the name of the entity (e.g “contact”, “account”, “restaurant”), the record identifier, and which columns to retrieve. In my map, I’ve set the AllColumns node to true which means that everything comes back. Otherwise, I’d need some custom XSLT in a functoid to populate the Columns node.

Mapping the Create Message

The “create” message is more complicated as we need to successfully build up a set of name/value pairs. Let’s walk through the steps.

The first “page” of my map links the entity’s name and sets a few unused elements to null.

Now it gets fun. You see a node there named KeyValuePairOfstringanyType. This node is repeated for each column that I want to populate in my created Entity. I’m going to show one way to populate it; there are others. On this map page, I’ve connected each source node (related to a column) to a Looping functoid. This will allow me to create one KeyValuePairOfstringanyType for each source node.

Got that? Now I have to actually map the name and value across. Let’s break this into two parts. First, I need to get the node name into the “key” field. We can do this by dragging each source node to the “key” field, and setting the map link’s Source Links property to Copy Name. This copies the name of the node across, not the value.

So far so good. Now I need the node’s value. You might say, “Richard, that part is easy.” I’ll respond with “Nothing is easy.” No, the node’s name, KeyValuePairOfstringanyType, gives it away. I actually need to set an XSD “type” property on the “value” node itself. If I do a standard mapping and call the service, I get a serialization error because the data type of the “value” node is xsd:anyType and Dynamic CRM expects us to tell it which type the node is behaving like for the given column. Because of this, I’m using a Scripting functoid to manually define the “value” node and attach a type attribute.

I also built an orchestration that calls the service and spits the result to disk, but there’s not much to that. At this point, I deployed the solution.

Configuring the Send Port

Now within the BizTalk Admin Console, I imported one of the bindings that the WCF Service Consuming Wizard produced. This makes life simple since there’s virtually nothing you have to change in the BizTalk send port that this binding produces.

The WCF-Custom adapter uses a custom WCF binding.

The only thing I added was on the Credentials tab, I added my Windows credentials for calling the service. After creating the necessary receive port/location to pick up my initial file, send port to emit the service result to disk, and bound my orchestration, I was ready to go.

Executing the Query

In my Dynamics CRM environment, I added a customer account record for “Contoso”. You can see a few data points which should show up in my service result when querying this record.

After calling the “Query” operation, I can see the result of the service call. Not particularly pretty. In reality, you’d have to build some mapping between this result and a canonical schema.

As for creating the record, when I send my command message in to create a new record, I see the new (Fabrikam) record in Dynamics CRM and a file on disk with the unique identifier for the new record.

Summary

So what’s “good”? Dynamics CRM 2011 is an excellent application platform for building relationship-based solutions and has a wide range of integration options. The REST interface is great and the SOAP interface will be useful for those that can leverage the CRM SDK. What’s “bad”? I don’t like the untyped interface. I know it makes future flexibility easier (“add an attribute to an entity, don’t change the interface!”), but it really handicaps BizTalk and other tools that can’t leverage their SDK components. I can’t see that many people choosing to build these functoid heavy maps just to create key/value pairs. I’d probably opt to just use a custom XSLT stylesheet every time. What’s “ugly”? Not thrilled with the shape of the software, from an integration perspective, this close to general release. Adding a simple WCF service reference to a .NET app should work. It doesn’t. Generated BizTalk schemas should be valid XSD. They aren’t. I don’t like the required “typing” of a node that forces me to do custom XSLT, even on a simple mapping.

I suspect that we’ll either see partner solutions, or even Microsoft ones, that make the integration story from BizTalk a tad simpler. And for all I know, I’m missing something here. I’ve vetted my concerns with the Microsoft folks, and I think I’ve got the story straight, however.

Thoughts from you all? Are you a fan of untyped interfaces and willing to deal with the mapping sloppiness that ensues? Other suggestions for how to make this process easier for developers?

Totally agree on the clunkiness of integration for anything other than a .NET 4-based application that can leverage the generated classes and proxy to do early binding. Even if you’re willing to deal with generic entities, communicating with the Organization service from something like Silverlight is a total nightmare right now, although MS has promised the Basic HTTP binding will be cleaned up in the RTM release. For the project I’m working on we’ve basically had to take the “wrapper” approach you describe, where we wrote our own WCF services to sit on top of the early-bound calls to CRM, and we do a fairly simplistic mapping to classes that actually expose a real data contract, to make for easy handling of data binding and such on the client side. Too bad this leaves yet another layer to maintain any time changes get made to the CRM data model.
It’s definitely a promising product for the XRM/app “building” scenario though. I’m pretty impressed with the capability there. I’d be shocked if they don’t improve the integration story in subsequent releases or add-ons.

Thanks for providing this example, Richard. A nice, real world, introduction to integrating with the latest version of Dynamics CRM. I’m kind of surprised that they are moving to a RESTful API. Is this a pattern with MS products?

Nice article. Thank you for post.
I would like to know few things more.
1. Whether you are using local installed version for integration or online instance.
2. How to pass user credential to wcf service if both CRM server and BizTalk are not in same domain and BizTalk user is not in same Active Directory where CRM user exists.
3. In case of accessing CRM online how you handle authentication as No place to specify CRM token to message in BizTalk.
It will be great if you could provide more detail. I need these information as i am working on integration of CRM 2011 online and BizTalk 2010. But i am stuck with Message Security Exception and unable to proceed without involving .Net code.
Some more detail regarding issue is at “http://social.msdn.microsoft.com/Forums/eu/crmonline/thread/44fb9a46-5f07-40a8-8b20-a3f3f90d070b”

1. Using a local (internal network) instance, not online
2. I had a big issue with this, but apparently things were corrected in the latest release candidate. I was building such examples myself, and while I got it working in straight .NET code, I couldn’t get it right from BizTalk.
3. I’ve seen some examples online of integrating with online, and suspect that a proxy service is going to be the best choice.

Hi Richard,
I came across your blog through search.. I am an architect for an early adopter of CRM 2011 , Like you mention in the post , the product is fantastic – we are very happy with all the changes in the product especially the new REST OData API (although it is restrictive) and the integration with Azure AppFabric, Charts , custom activities and so on..

I wanted to point out that we ran into similar difficulties with RC , but i was told that they are fixed with the final release. Overall, adding a reference to the WCF service from other tools works like a charm – we use it for one of our internal integrations, You can also see examples in the SDK.. I was told that this issue is because of the way BizTalk interprets WSDL..Either way, I was given the schema files that did not have the type schema errors , You should ask your MS contact if they can give you an early copy. The schema is all static , so there should not be an issue with me posting it , let me confirm with the MS team.

There are other techniques to get early binding as well – So i do not agree with some of your conclusions.Feel free to ping me at davidfo@live.com

Thanks for the thoughts, David. I have confidence that quirks just as consuming services would be ironed out by the final release. Glad that you can provide confirmation of this.

To which other early binding techniques do you refer to? Is there another option for BizTalk consumption of the service? While I could convert early bound classes to XSD schemas for use in BizTalk (maybe), I still need to build something to convert to the base entity and back, no?

Microsoft finally listened. I told them Visual Studio/WCF-style stuff Axapta won’t work with the BizTalk as of 2006 version. A very different approach. So, now they are re-writing WCF under a different labe.

Hey Ben. Yes, the new SDK has schemas. The README for the schemas says:
“These schema files should be used to develop Microsoft BizTalk applications that interface with Microsoft Dynamics CRM. For example, these schema files can be used with the out-of-box Microsoft BizTalk WCF adapter. Currently, these schema files are required because connecting to the Microsoft Dynamics CRM endpoints directly from within a BizTalk project does not result in the schema files for the endpoint being generated correctly.”

i’ve done a lot of CRM integration from BizTalk before and it’s always been a relatively simple process. i’ve never really used the adapter before, always just used WCF directly (with a custom behavior to deal with the additional MSCRM4 header). Always had access to strongly typed entities before.

seems bizarre that they’ve dropped this.
is it official that there will be no further CRM adapter? Probably cause they now want to push their OOB Dynamics connector tool.

perhaps there’s a market for someone to build a decent WCF-LOB adapter!

Jonathan, I usually cheat and look at the orchestration that gets generated by the generated service reference. In that orchestration you’ll find a series of port types which in turn point to the message types that each operation expects.

In terms of what should be there, it was a bit of trial and error. My newly announced book includes a chapter on CRM 2011 integration which hopefully clears this up further.

Hey there John. Agreed that this map will likely get unsustainable after a while. A custom pipeline would be needed on both sides (send/receive) of a send port in many cases. You need to transform the outbound content (e.g. create “account”), and in some cases, the inbound content as well (“account” returning from a query).

Joon, they are only used for late binding as that’s all that works with BizTalk right now. You can directly add them to your project and don’t HAVE to do the Add Generated Items process. However, you may still want to run that wizard JUST to get the binding files.

What I know about CRM 4.0 WCF adapter is that they provide you a schema called “CrmService_schemas_microsoft_com_crm_2007_WebServices.xsd” where you can find out the exact entity name and its all attribute lists. You need to pass SOAP header to that message with Organiztaion name, Authentication type etc.

But it is completely different in CRM 5.0 WCF Adapter schemas. There are benefits for UnType messaging, but there are additional actions to be performed (which I assume well suited for well educated people only).

Well to be very frank, I tried to search the WCF Schemas where I can map my data into and it took me some time to find out(“organizationservice_schemas_microsoft_com_xrm_2011_contracts_services.xsd”).

your post was a great help. But I have one problem left:
How to set lookup values?
With the BizTalk Adapter of CRM 4.0 I’ve just inserted the ID of the related entity and it worked. Now nothing happens…

got it!
I think it’s ugly but it is a solution that works 😛
I’ve set the type of the referencing value-node to EntityReference and added two additional nodes (Id and LogicalName) in the Scripting Functoid.
Now the Lookup gets set.

I have an issue that I am not able to resolve. I am tying to update an OptionSet field in CRM and I get deserialization error if use xs:string type in my xslt in the map. What type should I use for OptionSet?

I tried your suggestion, I am getting the following error. I also bought your book and tried the method specified in that, i did not get back any error, but the optionset field never gets updated.

–
Sender
–
a:DeserializationFailed

–
The formatter threw an exception while trying to deserialize the message: There was an error while trying to deserialize parameter http://schemas.microsoft.com/xrm/2011/Contracts/Services:entity. The InnerException message was ‘Error in line 1 position 939. Element ‘http://schemas.datacontract.org/2004/07/System.Collections.Generic:value’ contains data from a type that maps to the name ‘http://www.w3.org/2005/08/addressing:OptionSetValue’. The deserializer has no knowledge of any type that maps to this name. Consider changing the implementation of the ResolveName method on your DataContractResolver to return a non-null value for name ‘OptionSetValue’ and namespace ‘http://www.w3.org/2005/08/addressing’.’. Please see InnerException for more details.

Hi Anand,
I’m trying to figure out the same thing. I tried using ns4:OptionSetValue as type and an additional Value-element (not the value element of the key-value pair).
With that I get the error “A validation error occurred. The value of ‘fieldxy’ on record of type ‘entityxy’ is outside the valid range”.
I doubled checked if the OptionSetValue that I pass to the service – it really exists.

Update: I had to use the namespace prefix also for the Value element. OptionSetValue is now set correctly. I wish it would be that hard to do such a simple task. For me the new appraoch is a step backwards.

unfortenatly xml can’t be inserted in the comments… What I ment was that you have the key/value elements and under the element “value” you need another element “Value” that contains the actual integer value of the optionlistvalue.

great post!
with an “on premise” installation of crm it works like a charm!
Then i tried to do it with crm 2011 online. so far so good…

but i got a problem. when i use the binding-file which is created by visual Studio (with credentials off course…) i get a fault message that says: “An error occurred when verifying security for the message”. i`ve googled and found a lot of pages that states the problem is a time synch problem. but i dont have influence on the cloud server (how could i *g*)
did you edit the binding info?
because i dont have a security node in the binding-tab.

Hi Richard,
we could not succeed yet in creating a new account. Please see the input XML and the Output error message. We have tried “everything”…
And what is the purpose of the “LogicalName”?
We would appreciate your help very much.

This article help me a lot on my first steps with CRM integration with BizTalk. Thanks for that.

Right now I’m facing a big issue that, in the comments, it was already discussed but still no solution.

”
2. How to pass user credential to wcf service if both CRM server and BizTalk are not in same domain and BizTalk user is not in same Active Directory where CRM user exists.
”

I already tried the wcf-custom with lot of customization (many sites talk about this, but no solution). Can you, or someone, help me on this?! Always getting the frustrating message “The caller was not authenticated by the service”.

This is a great article and certainly helped get me started, but I think I have missed something. My task is to get a message from CRM 2011 into biztalk, I have the schema for how I want it to appear, I have consumed the service and replaced the schemas from the SDK, but now I am completely stuck. Your example shows your message (DemoInput.xsd) being mapped for consumption by CRM, what I need is to map from CRM into my xml format.

So you’re going from CRM to BizTalk. So the SDK schemas shouldn’t come into play since you are just creating a BizTalk-generated endpoint and consuming it from a CRM component (plug-in, workflow, etc). Is that right? Or are you trying to pull (vs push) from CRM into BizTalk? For the push-from-CRM-to-BizTalk scenario, I did write about this exactly scenario in the recent book.

Thanks Richard, have got a copy of your book which I will read with much interest. I have to admit to knowing nothing about CRM but am assuming that it will push the message. I also re read and had a bit of an aha moment about the KeyValuePairs and how that relates to the actual fields in CRM, so hopefully with your book in hand it will all fall into place, thank you for getting back to me

replacing the schemas generated by the “add generated item” with those in the CRM SDK (5.0.10) invalidates all the multipart message types created in the “organizationservice.odx”… and I had to manually reassign the type for each multipart message. 😦

The issue seems to stem from difference in filenames generated by the “add generated item” wizard and those that exist in the SDK. The ones generated by the wizard has mixed case like “OrganizationService_……._Metadata.xsd” while those in the SDK are all lowercase. Setting them to the correct case and importing them into the projects set things right.

I am getting an error while adding generated items through Consume WCF Service wizard.

When I enter the service URL and press get button it gives the following error.

Metadata not available
Failed to get metadata from “https://mycrm/myorg/XRMServices/2011/Organization.svc”.

(Microsoft.BizTalk.Adapter.Wcf.Consuming.MetadataExchange.MetadataExchangeException) Unable to download metadata from “https://mycrm/myorg/XRMServices/2011/Organization.svc” using WS-Metadata Exchange. (System.InvalidOperationException) Metadata contains a reference that cannot be resolved: ‘https://mycrm/myorg/XRMServices/2011/Organization.svc’. (System.ServiceModel.Security.MessageSecurityException) The HTTP request was forbidden with client authentication scheme ‘Anonymous’. (System.Net.WebException) The remote server returned an error: (403) Forbidden.

CRM is configured using claim based authentication over IFD

When I add the URL
“https://mycrm/myorg/XRMServices/2011/Organization.svc?wsdl”. and press get button it shows the wsdl but when I press next

it gives object reference not set to an instance of an object error and it does not generate bindings and schemas.

However I have add a service reference in another test project and imported the app.config file from this project to the send port import configuration.

When I enable the send port after the configuration it gives following error

Error details: System.ServiceModel.Security.MessageSecurityException: An unsecured or incorrectly secured fault was received from the other party. See the inner FaultException for the fault code and detail. —> System.ServiceModel.FaultException: ID3242: The security token could not be authenticated or authorized.
— End of inner exception stack trace —

Receive Port (receive from local folder) – XMLReceive
Sent Port (WcfSendPort_OrganizationService_CustomBinding_IOrganizationService_Custom) – both send and receive pipeline ‘PassThuruTransmit’ (if i set receive pipeline XMLReceive, the biztalk not retrieve any record from CRM, even it not passing send request step)
Sent Port (send to local folder) – ‘PassThuruTransmit’

I just created simple project to retrieve from crm. i place the xml in receive place

FE97FE5B-EAA2-E211-AC12-0050568C2EE8

it works up to receive (Receive Result) steps (retrieve from CRM ), but not output placed in the send port, giving the below error

Uncaught exception (see the ‘inner exception’ below) has suspended an instance of service ‘Account.OrganizationServiceClient(2111123a-fc05-2930-5095-8be7f08ae4b5)’.
The service instance will remain suspended until administratively resumed or terminated.
If resumed the instance will continue from its last persisted state and may re-throw the same unexpected exception.
InstanceId: 20792407-795f-471f-84cc-73fbadf35573
Shape name:
ShapeId:
Exception thrown from: segment -1, progress -1
Inner exception: Received unexpected message type ” does not match expected type ‘http://schemas.microsoft.com/xrm/2011/Contracts/Services#RetrieveResponse’.

Exception type: UnexpectedMessageTypeException
Source: Microsoft.XLANGs.Engine
Target Site: Void VerifyMessage(Microsoft.XLANGs.Core.Envelope, System.String, Microsoft.XLANGs.Core.Context, Microsoft.XLANGs.Core.OperationInfo)
The following is a stack trace that identifies the location where the exception occured

That typically sounds like you’re receiving a fault message back from CRM, not the expected message type. When you have tracking in place, can you see exactly what response you’re getting from Dynamics?

Finally after a long but interesting journey i have arrived at “Using a Dynamics CRM proxy service with BizTalk Server” section of your great book….unfortunately i have hit a wall.
keep getting the following two errors : please help
The Messaging engine failed to process a message submitted by adapter:FILE Source URL:C:\Integration CRM Solution\Receive\*.xml. Details:The published message could not be routed because no subscribers were found. This error occurs if the subscribing orchestration or send port has not been enlisted, or if some of the message properties necessary for subscription evaluation have not been promoted. Please use the Biztalk Administration console to troubleshoot this failure.
AND
A message received by adapter “FILE” on receive location “CRM2011.PickupCustomerQuery.FILE” with URI “C:\Integration CRM Solution\Receive\*.xml” is suspended.
Error details: The published message could not be routed because no subscribers were found. This error occurs if the subscribing orchestration or send port has not been enlisted, or if some of the message properties necessary for subscription evaluation have not been promoted. Please use the Biztalk Administration console to troubleshoot this failure.
MessageId: {4FE276AB-CF21-4450-BDFA-118C5828A1CB}
InstanceID: {2EACBEFE-E9DB-44CE-9B2C-4B573DD2FF2C}
regards

Hi Richard, so those two messages pretty clearly state the error. Are you sure that you have something listening for that inbound message? Are you using an XMLReceive pipeline that promotes the message type, or routing based on some other property?

Dear Richard. thank you for the response.
i am using xml receive on the receive port. i used tracking on the on the generated send port. there is a transmission failure. the data is passed on to the add customer scheme and goes no further. will redo the section to see if i get different results.

I’m afraid I’m facing a little trouble here with that ‘anytype’ node in the criteria section of service ‘RetrieveMultiple’.

I’m trying to build a query that retrieves all the accounts in an account guid set

accountid
In

$whateverguid

I don’t know what could be wrong, but from this message (Object reference not set to an instance of an object.
) i guess something is malformed in the Values Node. Could anyone help me on this, please?