Friday, February 27, 2009

The Visual Studio 2008 Profiler is....

in nice words....

complete crap.

I just upgraded my Visual Studio Professional 2008 Version to a 2800 USD Team Edition. Of course I installed SP1 first. The reason behind was that our WCF Service project is getting to a point where performance is getting relevant, so we needed to get some profiler results on our code.

So if you think you can install Visual Studio Team or Developer Edition and just hit a "Profile Code" button and get some results you are wrong.

For some reason the profiler does not collect any data

"PRF0025: No data was collected"

is all I got the first day. No matter whether I used the "Sampling" or "Instrumentation" method. All I got was straight nothing.

The documentation seems to be close to nothing as well. But then I stumbled into some web page and found out that the whole profiler thing is located in

%PROGRAMFILES%\Microsoft Visual Studio 9.0\Team Tools\Performance Tools

There is a little batch called VSPerfCLREnv.cmd which then can called with various options.
But when you check out this little tool you find out that the whole profiler thing is based on enviroment variables. And guess what you have to do if you set one of the /global parameters...

Yes reboot. Every time.

After that at least could do some profiling with Instrumentation method.

Linq/Lambda Expressions are not handled appropriately

But if the profiler did not collect any data before, now a single service function call collected more than 600MB of data. That's quite a lot if you think that the service function has not more than 100 lines of user code.

It took a while to find out what happened.


foreach (var item in items)
item.StaticInstrument = fullStatics.Find(fs => fs.ID == item.StaticInstrument.ID);

results in 600MB profiler data.

Rewriting the code to use a dictionary instead the List extension method Find(this List)

var itemDic = new Dictionary<string, StaticInstrument>();

fullStatics.ForEach(singleStatic => itemDic[singleStatic.ID] = singleStatic);

items.ForEach(item => item.StaticInstrument = itemDic[item.StaticInstrument.ID]);

results in 2MB of profiler data.

Functions are not displayed appropriatly in the reports.

But then the next disappointment is right behind the door.

In the report I have various multiple summaries like

Function Name:
0x2B000015
0x2B0000oE

This is not really helpful. Especially as the values for function called by the disguised functions are not displayed at all. Everything just gets summed up.

I took me another half day to find out what is the reason behind this.

In the above mentioned directory you find a tool called
VSPerfReport.exe

if you pass call it like

VSPerfReport.exe PATHTO\Report.vsp

I at least got

Warning VSP2701 : Symbol Engine: File '.....dll' could not be found when looking for debug information. Full symbol/type resolution may not be possible

But when you look at the location there is of course the DLL's symbol or pdb file right next to the dll. Why it still fails to load it, I don't know. I did not find any resolution for this problem yet.

So for most of DLL's the report fails to load the symbols. The resulting report is really useless.

I tried to write some mails to the guys from the Microsoft Profiler Team but it took a while to get some answer. See the comment for some suggestions from Marc Popkin-Paine. Unfortunately his suggestions don't work for me and meanwhile the profiler refuses again to collect any data at all.

Here is what happens...

Successfully attached to process: 4828
VSPerformance Automation warning: Warning VSP2348 : 2 buffer(s) lost in association with process 4828.
Exited from process: 4828
Collection file exited: C:\Development\WCFServices\SCDService\SCDServices090305.vsp
Profiler exited
PRF0025: No data was collected.

So if you ever thought about about spending money on Team or Developer Edition to use the profiler.... Save the money. The profiler is complete crap. You even have to be careful with it. I might crash your system with a blue screen of death. The reason behind this is that they obviously access the bios to get the exact number of clock cycles. Because of the different BIOS this is the reason, why it sampling does work for a certain number of systems. But this raises another question. Why the heck can a user process in Windows still read/write the bios and crash the system?

So in short...

With the Microsoft Profiler I wasted a lot of money and time without getting any useful results.

I have evaluated the current ANTS 4.3 profiler. ANTS works right out of the box and here I can see all functions but the disadvantage is a huge overhead. ANTS tries to eliminate the overhead in the calculation but is not totally succesful with this task.

2 comments:

elLoco said...

Here is the reply from Marc Popkin-Paine

"Hey,

Thanks for contacting us and sorry for the slow reply.

We'd like you to try a couple of things to determine the nature of the error you're running into.

Can you try rebuilding after deselecting "exclude small funcs" to ensure that the changes are picked up? Can you make sure that "exclude small funcs" is disabled for path the target project and the profiler session? The checkbox should be ignored if disabled in either location but we want to make sure.

Can you try using reflector or ildasm to ensure that the three functions a(), b(), and c() indeed exist in your binary and haven't been optimized away?

Feel free to post future questions to our forum (http://social.msdn.microsoft.com/forums/en-US/vstsprofiler/threads/) so that if we are able to track down the cause of your problem, other customers can search for this information afterwards. Thanks for using the VS profiler and let me know if I can help with anything else.

Marc Popkin-Paine
Visual Studio Profiler

Daryush Laqab said...

Hello,

Sorry we did not respond to your query as quickly as desired, and sorry for the poor experience with Profiler. I do apologize for that. Like Marc, I do encourage you to visit our forum at http://social.msdn.microsoft.com/forums/en-US/vstsprofiler/threads/. We constantly monitor our forum and have good participation from the community, so you will definitely have more eyes looking at the problem. I am glad that you and Marc have now connected. I have read through your communication to date with Marc. I am confident that you will be able to work through all the issues. I would like to add a few very general suggestions; maybe it will help.

First, it sounds like you had some trouble getting the profiler to collect data. There are a few reasons why you might see the PRF0025 message. If you are profiling a windows service, you may have to specify the /CS and /USER flags to ensure data is collected across processes (see this blog, or the MSDN doc). If you are profiling a GAC-ed assembly, you would need to ensure that the instrumented assembly is placed in the GAC after it is re-signed.

Second, LINQ/Data Expressions appear not to be handled appropriately. The Profiler gathers information about all the function calls in an application. Modification in how the application code utilizes the framework can change the number and pattern of function calls. You can get more information about LINQ performance from the forums on the Data Platform Development or C# / VB / Managed C++ Language.

Third, Functions are not displayed appropriately in the report. What you see here is the side effect of static instrumentation. 0x02….. function name indicate an indirect call (i.e. delegate and function pointers) ; at instrumentation time we do not know what the other side is. If you instrument the function that is being called by the indirect call, you will see the targeted function underneath the 0x2b000001 number in the calltree. We have filed a bug to address this confusing behavior.
Also, profiler has to be able to locate the PDB to resolve the symbols. Within Visual Studio IDE, profiler reports the binaries/assemblies it cannot locate symbols for in the Output window. “Performance” needs to be selected from the dropdown list titled “Show output from”.

You also mentioned that the profiler might crash your system.Tthere was a bug within profiler (VS 2005 and VS 2008) which prevented these versions from executing correctly on newly released Intel I7 processors, which we have fixed. Actually, the problem turned out to be related to a reserved bit field we were accidentally setting deep in our driver code. It was benign in previous chips, but in this newly released chip, it caused a failure. Fixes are available here: [1], [2], [3].



Thank you for using Visual Studio Profiler.

Daryush Laqab
Program Manager
VSTS Profiler, Code Coverage, and Test Impact Analysis
dalaqab@microsoft.com