Tuesday, June 3, 2014

Garbage Collector and Memory Fragmentation

Sometimes I've memory problems... Sorry, this is another story.
In .net sometimes we can have memory allocation problems.

If you allocate too many objects you can fragment heap memory.
Though you have sufficient free memory the system could not be able to find a block of memory to use for allocation request.
In this case you obtain "System.OutOfMemoryException" though there is still available memory.

This scenario can occur when you have WPF application or when you need to load a large object when memory is fragmented due to loading and unloading objects.

If you have those problems the class below is for you.
In your application you can call MemoryTestApp.MemoryUtils.CompactMemory();
With this method Garbage collector is called (just in case) and are called Windows API to "compact a specified heap by coalescing adjacent free blocks of memory and decommitting large free blocks of memory"
The main API called are:

  • SetProcessWorkingSetSize
  • HeapCompact

With first API we ensure that the working size in memory allocation is set to the minimum.
With second API we compact all allocated block of memory.

First we say to the system that we want the minimum page size when one object is allocated to the heap, second we compact all the allocated memory's pages.


using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Linq;
using System.Runtime.InteropServices;
using System.Text;
using System.Threading.Tasks;

namespace MemoryTestApp
{
    /// <summary>
    /// Utilities to free anche compact memory
    /// </summary>
    public static class MemoryUtils
    {
        [DllImport("kernel32.dll")]
        [return: MarshalAs(UnmanagedType.Bool)]
        private static extern bool SetProcessWorkingSetSize(
            IntPtr process,
            UIntPtr minimumWorkingSetSize,
            UIntPtr maximumWorkingSetSize);

        [DllImport("kernel32.dll", SetLastError = true)]
        private static extern IntPtr GetProcessHeap();

        [DllImport("kernel32.dll")]
        [return: MarshalAs(UnmanagedType.Bool)]
        private static extern bool HeapLock(IntPtr heap);

        [DllImport("kernel32.dll")]
        private static extern uint HeapCompact(IntPtr heap, uint flags);

        [DllImport("kernel32.dll")]
        [return: MarshalAs(UnmanagedType.Bool)]
        private static extern bool HeapUnlock(IntPtr heap);

        private static void SetProcessWorkingSetSizeToMin()
        {
            SetProcessWorkingSetSize(
                Process.GetCurrentProcess().Handle,
                (UIntPtr)0xFFFFFFFF,
                (UIntPtr)0xFFFFFFFF);
        }

        /// <summary>
        /// Call Garbage collector, wait for it and Compact heap memory
        /// </summary>
        public static void CompactMemory()
        {
            CompactMemory(true, true);
        }
        /// <summary>
        /// Compact heap memory
        /// </summary>
        /// <param name="gcCollect">If true call GC.Collect</param>
        /// <param name="waitCollect">If true wait for GC.Collect</param>
        public static void CompactMemory(bool gcCollect, bool waitCollect)
        {
            if (gcCollect && waitCollect)
            {
                GC.Collect(GC.MaxGeneration);
                GC.WaitForPendingFinalizers();
            }
            else if (gcCollect && !waitCollect)
            {
                GC.Collect(GC.MaxGeneration);
            }

            SetProcessWorkingSetSizeToMin();
            HeapCompact();
        }

        private static void HeapCompact()
        {
            IntPtr heap = GetProcessHeap();

            if (HeapLock(heap))
            {
                try
                {
                    if (HeapCompact(heap, 0) == 0)
                    {
                        //error ignored
                    }
                }
                finally
                {
                    HeapUnlock(heap);
                }
            }
        }

    }
}

Saturday, May 3, 2014

Trace WCF

Very Often people ask to me how to configure web.config and app.config of .net website and .net application to emit trace information about WCF communication. So every time I search in my projects those settings.
To avoid this search I put in this blog those settings.

app.config
Below there is a portion of app.config that activate two trace and generate two file.
client_messages.svclog contains trace about messages exchanged between client and server.
client_traces.svclog contains trace about detailed information of System.ServiceModel Activities. 

Principal config sections are: <system.diagnostics> and <diagnostics>.
<system.diagnostics> section is child of  <configuration> element and in this example is at the same level of <appSettings>
<diagnostics> section is child of <system.serviceModel> element that is child of   <configuration> which is at the same level of <appSettings> and <system.diagnostics>

  ...
  </appSettings>
  ...
  <system.diagnostics>
    <sources>
      <source 
        name="System.ServiceModel.MessageLogging"           
        switchValue="Verbose">
        <listeners>
          <add 
            name="xml"       
            type="System.Diagnostics.XmlWriterTraceListener"
            initializeData="c:\temp\client_messages.svclog"/>
        </listeners>
      </source>
      <source 
        name="System.ServiceModel" 
        switchValue="Verbose" 
        propagateActivity="true">
        <listeners>
          <add 
            name="traceListener"
            type="System.Diagnostics.XmlWriterTraceListener" 
            initializeData="c:\temp\client_traces.svclog" />
        </listeners>
      </source>
    </sources>
    <trace autoflush="true"/>
  </system.diagnostics>

  <system.serviceModel>
    <diagnostics>
      <messageLogging 
        logEntireMessage="true"    
        logMalformedMessages="true"
        logMessagesAtServiceLevel="true"
        logMessagesAtTransportLevel="true"
        maxMessagesToLog="1000" />
      <endToEndTracing 
        activityTracing="true"
        messageFlowTracing="true" />

    </diagnostics>
  </system.serviceModel>

web.config
Below there is a portion of web.config that activate two trace and generate two file.
server_messages.svclog contains trace about messages exchanged between server and client.
server_traces.svclog contains trace about detailed information of System.ServiceModel Activities. 

Principal config sections are: <system.diagnostics> and <diagnostics>.
<system.diagnostics> section is child of  <configuration> element and in this example is at the same level of <system.web>
<diagnostics> section is child of <system.serviceModel> element that is child of   <configuration> which is at the same level of <system.weband <system.diagnostics>


  ...
  </system.web>
  ... 
  <system.diagnostics>
    <sources>
      <source 
        name="System.ServiceModel.MessageLogging" 
        switchValue="Verbose">
        <listeners>
          <add 
            name="xml" 
            type="System.Diagnostics.XmlWriterTraceListener" 
            initializeData="c:\temp\server_messages.svclog"/>
        </listeners>
      </source>
      <source 
        name="System.ServiceModel" 
        switchValue="Verbose" 
        propagateActivity="true">
        <listeners>
          <add 
            name="traceListener" 
            type="System.Diagnostics.XmlWriterTraceListener"                 initializeData="c:\temp\server_traces.svclog" />
        </listeners>
      </source>
    </sources>
  </system.diagnostics>

  <system.serviceModel>
    <diagnostics>
      <messageLogging 
        logEntireMessage="true" 
        maxMessagesToLog="1000" 
        logMessagesAtServiceLevel="true"        
        logMalformedMessages="true" 
        logMessagesAtTransportLevel="true"/>

    </diagnostics>
  </system.serviceModel>

Note 
Notate that every file has ".svclog" as file name extension.
In this way the file are opened automatically by Microsoft Service Trace Viewer.

Friday, May 2, 2014

WCF messge size tips and tricks

The default size of message in SOAP\WCF communication is 65536 bytes.

Sometimes you may need to send soap message bigger than 65536 bytes.

If you exceed that limit you obtain this exception:
System.ServiceModel.ProtocolException occurred
  Message=The remote server returned an unexpected response: (413) Request Entity Too Large.

To go beyond this limit you have to modify the service's binding configuration settings of web.config and app.config .

Setting are:
maxReceivedMessageSize of binding configuration
maxArrayLength of readerQuotas section
maxArrayLength of readerQuotas section
maxBytesPerRead of readerQuotas section
maxDepth of readerQuotas section
maxNameTableCharCount of readerQuotas section

All those setting require an int value, so the maximum value is the int.MaxValue.
Below there is an example of basicHttpBinding with all size values to the maximum value.

<basicHttpBinding>
    <binding
        closeTimeout="01:00:00"
        maxBufferPoolSize="2147483647"
        maxBufferSize="2147483647"
        maxReceivedMessageSize="2147483647"
        openTimeout="01:00:00"
        receiveTimeout="01:00:00"
        sendTimeout="01:00:00">
       <readerQuotas          
maxArrayLength="2147483647"          
maxBytesPerRead="2147483647"          
maxDepth="2147483647"          
maxNameTableCharCount="2147483647"                  
maxStringContentLength="2147483647"
        />  
</binding>
</basicHttpBinding>

In this way you can have soap messages with size about to 2gb.
But you must pay attention to do this because a single message can put your server in denial of service (DOS).
Size of 2gb is not a good idea.

All above said is formally corrected but... surprise surprise doesn't work completely. Yes of course.
If you are hosting your web service in IIS and your message is greater then 30,000,000 bytes, client side you'll receive this exception:

System.ServiceModel.EndpointNotFoundException occurred
  Message=There was no endpoint listening at http://localhost/WebApplication1/Service1.svc that could accept the message. This is often caused by an incorrect address or SOAP action. See InnerException, if present, for more details.

This exception is right because IIS enter in action before WCF "runtime" to filter requests and prevent denial of service (DOS) attacks.

If you need to go beyond 30,000,000 bytes you have to modify webserver's settings.
You can modify webserver's settings modifying web.config like the example below.

Setting to modify is maxAllowedContentLength that require integer value.

<system.webServer>
        <staticContent>
            <mimeMap fileExtension=".desk" mimeType="application/x-zip-compressed" />
            <mimeMap fileExtension=".crc" mimeType="text/plain" />
        </staticContent>
        <security>
            <requestFiltering>
                <requestLimits maxAllowedContentLength="2147483647" 

                   maxQueryString="2147483647" />
            </requestFiltering>
        </security>

    </system.webServer>

There are other ways to alter requestfiltering setting using "IIS Manager" described at this link:
http://www.iis.net/configreference/system.webserver/security/requestfiltering
Those ways at the end alter web.config.

Recap

  1. If you need to create a WCF service that accept more than 65536 bytes and less than 30,000,000 bytes of data you have to modify app.config and web.config altering maxReceivedMessageSize and readerQuotas.          
  2. If your messages are bigger than 30,000,000 bytes you have to alter all maxReceivedMessageSize, readerQuotas and requestFiltering settings with same values.
Pay attention because big amount of data can cause Denial Of Service problems, so be careful and apply this setting only on binding defined for the specified service that require so much data. 
Remember that for large amount of data scenario WCF support streaming transfer mode (but this is another story).