Thinking WCF towards Enterprise SOA

You want know how to build your own UDDI based SOA?

Read the UDDI SOA Howto.

Update: Get the article including source code at http://www.codeproject.com/KB/WCF/uddiservicefactory.aspx

Is WCF out of the box a Service Oriented Architecture?

If you think that WCF is SOA then lets review part of the definition of "Service Oriented Architecture".

OASIS (the Organization for the Advancement of Structured Information Standards) defines SOA as the following:

"A paradigm for organizing and utilizing distributed capabilities that may be under the control of different ownership domains. It provides a uniform means to offer, discover, interact with and use capabilities to produce desired effects consistent with measurable preconditions and expectations.

Now lets translate "capability" with "service".

So we get in slightly other words: "A SOA consists of distributed services, some sort of mechanism to offer and find them, a mechanism to interact with the service and finally a mechanism to invoke the service.

If you try to break down this into web service technology terms you now translate this into:

• Services (this is what WCF is all about)
• Find and Offer (this could be a UDDI Registry (Enterprise), or WS-Discovery (UPnP like))
• Interact (this could be WSDL)
• Invoke (this could be SOAP)

So what part of SOA is covered by WCF?
The service, the interaction and the service invocation. The runtime discovery is not part of WCF.

Conclusion:
WCF offers no proper mechanism to offer and find services at runtime and hence is a classical 3-tier architecture technology, that we basically have for 15-20 years now. We just called it DCOM or Corba or whatever before and now the same technology gets relabeled with the current buzzword "Service". Other examples I love are:
"SAAS" or "Software as a Service" what was called "ASP" or "Application Service Provider" before
"ESB" or "Enterprise Service Bus" what was called "EAI" or "Enterprise Application Integration" before
The thing about buzzwords is that you can throw them around with having no clue what they are really supposed to mean. So the words service in "Service Oriented Architecture" and "Web Service" are commonly misunderstood to be same where in fact they mean two different things.

Just a web service is not a SOA.

WCF follows Web Services

The WCF basically combines all the different communication technologies of .Net 2.0 (Remoting, Queues etc) under one roof and a common programming model. The model itself follows the web service model introduced with .Net 2.0. One of the biggest disadvantages of web services was that the communication layer for web services pretty much fixed to WS-I standard (BasicHttpBinding in WCF terms). What Microsoft now basically did for WCF was to address this issue and introduced a completely flexible communication layer. So instead of a preassigned WS-I protocol we can now choose from a widespread of communication protocols that nearly fit every situation.

But from a programmers point of view, the implementation of WCF services/clients follows pretty much the way we programmed web services. On the client side instead of a "web service reference" you now just add a "service reference" to your project but then it feels just like before. On the service side you now have to define a service interface and you use some other attributes but then again it feels like being with web services.

But this flexibility has some implications.

• With every service reference you now add a configuration file with a "ServiceModel" section that keeps all the protocol specific configuration options.
• The path to the WSDL for web services was pretty much always http://ServiceAddress?WSDL . Now with non http protocols there must be another way to access the Service Metadata. So with WCF the "Mex" or "MetadataExchange" endpoint, that is supposed to be on the "ServiceURL/mex" address, was introduced.

Target Enterprise Architecture

Guess the following scenario: I have 10 different types of Services running, with 100 clients each. An hour of unscheduled service downtime during day hours can cost me up to a 100 million bucks.

So what are my requirements?

• I must be able to relocate my services without any clients failing
• Services must be redundant and clients must support failover
  
• I must be able to dynamically add more services of same type for load balancing and fault safety
• I want to have a quick overview where my services are running
• I must be able to change communication parameters (e.g. enable transport encryption on the service, without any of my clients failing

Does standard WCF Implementations meet our requirements?
By "Standard" I refer to the way the the service and the client had been created. Means setting up the service, then creating a client, adding the service reference to the service and then calling the service function of interest.

What happens if we:

• Move the service to another server?
  The client fails.
• A service fails?
  The client fails.
  
• Change communication parameters of the service such as binding or encryption?
  The client fails.
  
• Want to introduce another service of the same kind for load balancing?
  Installed clients ignore it
  
• I want to get a quick overview where my services are running?
  I can't.

As a result for every above change I would need to update 100 hundred clients config files, but I have no clue on what workstations the clients are located. As a result, any above change breaks my production process, which is not acceptable in a enterprise environment.

Conclusion:
The standard implementation of WCF is not suited for use in Enterprise environments.

Introducing a central Service registry to our architecture

As a mechanism to find and offer web services in an enterprise environment this rings the bell for UDDI which is included since Windows Server 2003, but which seems to be fairly unnoticed so far. If I say documentation and support is sparse, this is quite a extenuation.

Web Services in a UDDI environment.
Handling web services in a UDDI registry environment was pretty much straight-forward. You simply stripped everything but the Types, Message and Operations sections out of your WSDL, published it as a model, and then you added the service endpoints to the registry (where there was just one per Service).
As a client, you added the web service reference to the stripped WSDL, then at runtime you simply put in the address from the UDDI and everything was supposed to work just fine as the communication layer was fixed to WS-I and there was just one endpoint.

WCF Services in a UDDI enviroment
With WCF we have a whole bunch of possibilities on how our service can communicate. This includes the option that a service has multiple endpoints offering different communication protocols at the same time. If we have multiple instances of a service we no longer can assume that they will all use the same communication protocols.

But if we try handle WCF Services with UDDI just like a Web services then what happens?

At first this means:
First we strip the everything but the Types, Message and Operations sections from our WSDL.
This is where you first time stop. Looking at the WSDL you will find out that the "Types" section is not declared incline, but every namespace just has a import reference.

You have 2 options now.

• You open all the imports URL's in a browser and then you replace the imports with the results (sucks like hell after the 10th time) or
• You modify the WSDL behavior of your service with e.g. FlatWSDL. But also this way is not perfect as it sounds as there is a bug in Visual Studio that does not let you choose your array types any more under some circumstances. So if your Service returns an array you could configure it to be a List in your code, but under some circumstances Visual Studio now will not let you change the array type to be anything but array. This is quite nasty as the WSDL is perfectly valid and the type is perfectly recognized as MyType [] but you just will not get is as List anymore, no matter what you select as your ArrayType.

But after we made a decision on this first problem now, what happens if you simple exchange the clients ServiceEndPoint adress with the URL you get from the UDDI Registry.

What happens if we:

• Move the service to another server?
  The client works if it does not use encryption, otherwise it fails.
• A service fails?
  The client works if it does not use encryption, otherwise it fails.
• Change communication parameters of the service such as binding or encryption?
  The client fails.
• Want to introduce another service of the same kind for load balancing?
  The client works if it does not use encryption, otherwise it fails.
• I want to get a quick overview where my services are running?
  I get anytime a nice overview from my registry.

So if we use any kind encryption the concept we used for web services fails all but one of our architecture requirements.

Can't WCF Services be integrated into a UDDI environment?

The problem is basically the "ServiceModel" section in our clients config file. It freezes the state of the service to the point where we added the service reference. If you use a binding with encryption which is by default based on Windows users you find for example a tag called "Identity" which is usually something like "Hostname\Username" and which is used for the clients encryption to the service. But once we move the service to another server this of course will fail due to the identity mismatch.

Integration of WCF Services into a UDDI environment

How to query the Service Metadata at runtime?
I remembered I had a little web service project where we dynamically queried a services WSDL, created a DOM object and the compiled the client at runtime. So my first approach to WCF was similar, and it works.
But then what I basically want to do is the more or less the same what the WCFTestClient.exe does. You simply put in a URL to a service, WCFTestClients discovers it and then creates a client. But doing a little little reverse engineering using Lutz Roeders excellent Reflector tool revealed that....

WCFTestClient.exe just invokes svcutil on the command line. :-))) . Who dared to ship a awkward solution like this?

But when I did the same with "svcutil" I stumbled across two classes in System.ServiceModel.Description namespace.

MetadataExchangeClient and MetaDataResolver

What both basically do is to read the services WSDL and then return a ServiceEndpointCollection. Now you just have to choose which endpoint you want to use for communication with your service at runtime.
But here the first questions start. What do I need MetaDataResolver for, that needs a MetadataExchangeClient and returns a ServiceEndpointCollection, when I can use the MetadataExchangeClients ImportAllEndpoints function? (Update: My Implementation is solely based on MetadataexchangeClient. I tried to save some lines of code by using MetaDataResolver, but it never worked out. So I just can discourage the use of MetaDataResolver.)

If you try get some light into the situation you will soon find out that in regards to WCF you now walk off the beaten track. Documentation and references will get very sparse from this point on.

But anyway. This is basically exactly what we need to do, if we want to integrate WCF Services into a UDDI environment. We get the main ServiceURL from the UDDI Registry, then we read the service metadata on ServiceURL/mex or ServiceURL?WSDL, then we pick an endpoint and then we call the desired service function.

So what we basically do is instead of freezing the services state to the point we add the service reference to our client project, we do the "Add Service reference" functionality every first time we want to talk to our service in our code! This basically means we do not need the <ServiceModel> in our config file anymore.

This works a while but then....
you will find out...

Why we needed the <ServiceModel> section
First thing going to crash is when you receive a service response that is larger than 64k. Your client will report that your "MaxReceivedMessageSize" quota is exceeded. And there quite frankly I ask myself the following question: Who the fuck decided that in the age of Tera- and Petabytes 64k would be a good default limit for receiving a service response? My electric toothbrush could handle a response larger than 64k. Why not 6k or even better nothing by default??? By the end of the project I am sure it will be more than a thousand times we had to adjust this pain in the ass default parameter. If you are moving on BasicHttpBinding you will need to modify the "MaxBufferSize" parameter as well. Interestingly this parameter has always to be the same as as "MaxReceivedMessageSize". A parameter that has always the same size like another parameter? Jeez...
The next superfluous default parameter modification that you will encounter is the "maxItemsInObjectGraph" that has the same arbitrary limit of 64k.

Ok, I could rant on for while but let's stop crying and look for a solution.

How to modify an unknown binding at runtime?

The core of the problem is that we do not know what bindings the service will offer.

MaxReceivedMessageSize and MaxBufferSize
Lets look at the bindings inheritance.
All Bindings are derived from class Binding but except from some timeout parameters there is nothing we can do here.
Lets look at the binding element stack.
In the stack of Binding Elements this parameter belongs to the Transport Binding Element, which is part of any Binding. But I tried 2 days to find a way to get access to the binding elements, but only for CustomBinding this will lead you somewhere. For all other bindings this is a dead end street. The CreateBindingElements() function will just create a copy of the actual bindings elements and the GetBindingProperty() function which comes along undocumented in official documentation is of no help either.

As we have no clue what binding the Service will offer we now have 3 options:

1. The IT-Hall of Shame candidate
   Write a function that tries to cast the discovered binding into any known binding and then sets the parameter
2. We use Reflection
   We use a function that first tries to cast the object into a custom binding. If the cast is successful it tries for every property of every binding element to set a property with "Name" to "Value" and if the cast fails just try to find a property with "Name" and then to set it to "Value".
3. We return a CustomBinding
   We use the discovered bindings CreateBindingElement() function, modify the parameter on the TransportBindingElement and return a CustomBinding instead of the original binding.

I don't like any of the solutions but solutions, but finally we decided to for the reflection solution.

How to modify MaxItemsInObjectGraph at runtime?
Interestingly "MaxItemsInObjectGraph" is a property of the DatacontractSerializer which is also used in all bindings.
If you use the config file you now have to add an Endpoint Behavior of type DataContractSerializer, but if you try to find in the ServiceModel namespace you will find out that you can not instantiate a this class at runtime. But looking further there is a DataContractSerializerOperationBehavior that can be instantiated at runtime but what can only be applied to a operation.
So what I did, was to create a class that implements IEndPointBehavior and then simply sets this DataContractSerializerOperationBehavior with a useful value to all operations.

Now I could get rid of my <ServiceModel> config section and all my functions worked just as intended.

So we got it?

So if you think this is enough struggling with 2 64k default parameters (yes I mean it sarcastically) you are soon to be disappointed.
Everything worked now, I could switch my service binding forth and back and my clients just worked fine until one day..... They stopped working.
MetadataExchangeClient reported that "Some items can not be resolved". It took me 2 days to find out what happened. Our data model had gotten bigger and bigger until one day.... Guess it.
Yes, the WSDL got larger than 64k. And this is the point where are all reference implementations of MetadataExchangeClient and MetadataResolver will fail. In case of MetadataExchangeClient even with a bolloks error message. And guess again. To solve the issue you have to create a Binding on your own, set this pain in the ass MaxReceivedMessageSize parameter and then use some constructors which are unmentioned in any example.

Finally!

This was so far the last submarine that submerged in front of my target architectures course. I don't say this was the last issue until of the end of the project, but yet I am confident that we will reach all architecture requirements with WCF without any limitations.

So what happens now if

• Move the service to another server?
  I change the address in UDDI and all my clients simply follow and continue working.
  
• A service fails?
  The clients does a nice failover and reconnects no matter what.
  
• Change communication parameters of the service such as binding or encryption?
  The client adopts to it.
  
• Want to introduce another service of the same kind for load balancing?
  Just register it to the UDDI and enjoy
  
• I want to get a quick overview where my services are running?
  I get anytime a nice overview from my registry.

All our target architecture goals are met.
Hooray! Awesome.
WCF implemention flaws

• The 64k default quotas on MaxReceivedMessageSize, MaxBuffersize and MaxItemsInObjectGraph and maybe elsewhere
  
• The default WSDL schema that uses "Import" statements in the "Types" section.
• The bug in Visual Studio that does not let you choose your ArrayType for flat WSDL's (MSConnect ID:387245)
• The inheritance of bindings that have highly redundant properties in the specific bindings and/or the lack of access to the binding elements for non "Custom Bindings"
• Inconsistencies between between config file configuration and runtime configuration
• Lack of documentation
  

Conclusion:

• WCF standard implementations are due to their "all is static" approach not Enterprise ready
  
• WCF Services can be integrated into a Enterprise SOA architecture but several implementation flaws prevent a seamless integration and increase project risk.

Suggestions to Microsoft for next WCF Release:

• Default binding quotas
  Quotas should be disabled by default. Any quota you set by default is arbitrary and will never fit all project scopes. If I do not trust the service, its nice that I can set this quota, but please let me choose whats best here. And especially do not use quotas that later blow up your own framework in a timebomb manner.
  
• "Imports" in WSDL by default
  This is only causing problems and has absolutely no use. Get rid of it.
• The inheritance of Binding
  Instead of a 2 level hierarchy it should be much more fine grained. e.g. in class BasicHttpBinding there should be only configuration properties that are absolutely unique to BasicHttpBinding. All others should be inherited. Adding more inheritance could be easily be done without breaking anything. This would smooth the way to a sophisticated runtime configuration of arbitrary bindings. I prefer this much more over a access to the binding elements as in CustomBinding.Elements.
• Runtime configuration and config configuration should be consistent
  

If there is some interest I may be able to publish some source code. Just leave a comment.