This post pulls the threads of those two older posts together and is really a runbook for my own future self: the companion for everything you find in a production dump that isn’t a plain leak — a hung request, a deadlock, an unhandled exception that took the process down, or a finalizer thread that stopped doing its job. It is a lookup table, ordered by symptom, and every command output below was produced on .NET 10 with dotnet-dump on Fedora — not copied from the docs.

Note: This post was put together with the help of an AI assistant

SOS is just WinDbg without the !

SOS is the debugger extension that teaches a native debugger about the CLR. On Windows you load it into WinDbg and call every command with a leading ! (!clrstack, !dumpheap). On Linux the same commands ship inside dotnet-dump, so you type them without the !.

Setup

Install the tool once (globally):

dotnet tool install --global dotnet-dump

On a minimal container image without ICU, dotnet refuses to start. Export DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=1 (or install libicu) before running any of the commands below. See Linux Manual installation and .NET globalization and ICU

Collecting a dump of a running process:

dotnet dump collect -p <PID> --type Full

For a process that crashes, let the runtime write the dump for you. Set these environment variables before starting the app and you get a dump at the exact moment of the unhandled exception, with no debugger attached:

export DOTNET_DbgEnableMiniDump=1
export DOTNET_DbgMiniDumpType=4          # 4 = full dump
export DOTNET_DbgMiniDumpName=/tmp/crash.dmp

Then open the dump:

dotnet dump analyze /tmp/crash.dmp

That drops you into an interactive > prompt. Every heading below is one command you run there. For scripting, pass commands with -c:

dotnet dump analyze crash.dmp -c "clrthreads" -c "pe -nested" -c "exit"

The sample program

All output below comes from one small console app that reproduces a scenario per command-line argument (lock, deadlock, crash, finalizer, threads) and prints its PID on startup. The interesting bits are shown inline with each section.

Download: Program.cs. To run it:

mkdir SosDemo && cd SosDemo
dotnet new console
curl -O https://www.stefangeiger.ch/assets/code/SosDemo/Program.cs
dotnet run lock # or: deadlock | crash | finalizer | threads

Threads & stacks

clrthreads lists every managed thread. The DBG column is the id you pass to setthread, OSID is the native thread id (hex), and the last column flags a thread that currently has a managed exception in flight.

> clrthreads
ThreadCount:      6
UnstartedThread:  0
BackgroundThread: 5
PendingThread:    0
DeadThread:       0
Hosted Runtime:   no

 DBG   ID     OSID ThreadOBJ           State GC Mode     GC Alloc Context     Domain           Lock Count  Apt Exception
   0    1     b63a 000055D119475610  2020020 Preemptive  ...                  000055D11944C600 -00001 Ukn
   5    2     b640 000055D119477D00    21220 Preemptive  ...                  000055D11944C600 -00001 Ukn (Finalizer)
   6    3     b641 000055D119480860    21220 Preemptive  ...                  000055D11944C600 -00001 Ukn
   8    4     b643 000055D1194CF160  2021220 Preemptive  ...                  000055D11944C600 -00001 Ukn
   9    5     b644 000055D1194DC6D0  2021220 Preemptive  ...                  000055D11944C600 -00001 Ukn
  10    6     b645 000055D1194DDDB0  2021220 Preemptive  ...                  000055D11944C600 -00001 Ukn

clrstack -all dumps the managed stack of every thread at once. This is usually the first thing I look at — it tells you what the whole process was doing:

> clrstack -all
...excerpt
OS Thread Id: 0xb643
        Child SP               IP Call Site
00007F8048DFD990 ... System.Threading.Monitor.Wait(System.Object, Int32)
00007F8048DFDA40 ... System.Threading.ManualResetEventSlim.Wait(Int32, System.Threading.CancellationToken)
00007F8048DFDAB0 ... System.Threading.ManualResetEventSlim.Wait()
00007F8048DFDAC0 ... SosDemo.Program.AcceptConnections()          [Program.cs @ 157]
00007F8048DFDAD0 ... SosDemo.Program.HttpListenerLoop()           [Program.cs @ 156]
00007F8048DFDAE0 ... System.Threading.Thread.StartCallback()

To dig into one thread, switch to it and print its stack:

> setthread 8
> clrstack

Lock contention

Symptom: requests hang, CPU is idle, one resource is a bottleneck. syncblk lists every SyncBlock that backs a CLR monitor (lock / Monitor.Enter) which is currently held:

> syncblk
Index         SyncBlock MonitorHeld Recursion Owning Thread Info          SyncBlock Owner
    1 00007F8230001958            7         1 00005650149D9220 b6e9   8   00007f827140c3d0 System.Object
-----------------------------
Total           1

Read it as: the monitor on object 00007f827140c3d0 is owned by thread b6e9 (DBG id 8). MonitorHeld is 2 * waiters + 1 when the lock is taken, so 7 means three threads are queued behind the owner.

pstacks groups threads by identical stack, which makes the contention obvious at a glance — one thread sleeps while holding the lock, three sit in Monitor.Enter:

> pstacks
...excerpt
      ~~~~ b6e9
         1 System.Threading.Thread.Sleep(Int32)
         1 SosDemo.Program+<>c.<LockContention>b__4_0()
      ~~~~ b6ed,b6ee,b6ef
         3 System.Threading.Monitor.Enter_Slowpath(Object)
         3 System.Threading.Monitor.Enter(Object, Boolean ByRef)
         3 SosDemo.Program.ChargeCustomer(Int32)

==> 5 threads with 2 roots

Now switch to the owner and find out why it is holding the lock for so long:

> setthread 8
> clrstack
        Child SP               IP Call Site
00007F8243FFEA10 ... System.Threading.Thread.Sleep(Int32)
00007F8243FFEAB0 ... SosDemo.Program+<>c.<LockContention>b__4_0()   [Program.cs @ 37]
00007F8243FFEAE0 ... System.Threading.Thread.StartCallback()

Deadlocks

A deadlock is just lock contention with a cycle. Here syncblk shows two held monitors:

> syncblk
Index         SyncBlock MonitorHeld Recursion Owning Thread Info          SyncBlock Owner
    1 00007FE550002578            3         1 000055C97DAC47C0 b70f   9   00007fe58940c3e8 System.Object
    2 00007FE5500025D0            3         1 000055C97DAB6970 b70e   8   00007fe58940c3d0 System.Object

Thread 8 owns object ...c3d0, thread 9 owns object ...c3e8, and MonitorHeld == 3 means one waiter on each. To see which lock each thread is blocked on, print the stack with arguments and locals (clrstack -a). The obj parameter of Monitor.Enter is optimised away, but the C# locals still hold the object references:

> setthread 8
> clrstack -a
...excerpt
00007FE55BFFEA80 ... System.Threading.Monitor.Enter(System.Object, Boolean ByRef)
00007FE55BFFEAA0 ... SosDemo.Program+<>c.<Deadlock>b__6_0()   [Program.cs @ 70]
    LOCALS:
        0x00007FE55BFFEAC0 = 0x00007fe58940c3d0     <- holds this one (SyncBlock 2)
        0x00007FE55BFFEAB0 = 0x00007fe58940c3e8     <- waiting for this one (SyncBlock 1, owned by thread 9)

> setthread 9
> clrstack -a
...excerpt
00007FE55B7FDAA0 ... SosDemo.Program+<>c.<Deadlock>b__6_1()   [Program.cs @ 79]
    LOCALS:
        0x00007FE55B7FDAC0 = 0x00007fe58940c3e8     <- holds this one (SyncBlock 1)
        0x00007FE55B7FDAB0 = 0x00007fe58940c3d0     <- waiting for this one (SyncBlock 2, owned by thread 8)

Thread 8 waits for a lock thread 9 owns, thread 9 waits for a lock thread 8 owns — the cycle is the deadlock.

Exception analysis (crash dumps)

With DOTNET_DbgEnableMiniDump=1 the dump is taken on the unhandled exception. clrthreads immediately points at the faulting thread — note the Exception column:

> clrthreads
...excerpt
 DBG   ID     OSID ThreadOBJ           State GC Mode     ...  Exception
   8    4     b7d0 000055EF9571ADE0    21020 Cooperative ...  System.InvalidOperationException 00007f2de940c9b8 (nested exceptions)

Switch to it and use pe -nested (print exception, including inner exceptions). This gives you the full chain with managed stack traces — usually everything you need:

> setthread 8
> pe -nested
Exception object: 00007f2de940c9b8
Exception type:   System.InvalidOperationException
Message:          Report generation failed for account 42
InnerException:   System.Collections.Generic.KeyNotFoundException, Use printexception 00007F2DE940C890 to see more.
StackTrace (generated):
    SP               IP               Function
    00007F2DC13F9E90 00007F3D7CE92BED SosDemo.dll!SosDemo.Program.RunReport()+0x8d
    00007F2DC13FDAE0 00007F3D7C0A9B65 System.Private.CoreLib.dll!System.Threading.Thread.StartCallback()+0x85
HResult: 80131509

Nested exception -------------------------------------------------------------
Exception object: 00007f2de940c890
Exception type:   System.Collections.Generic.KeyNotFoundException
Message:          Account 42 is not in the cache
StackTrace (generated):
    SP               IP               Function
    00007F2DC13FDA50 00007F3D7CE92CE9 SosDemo.dll!SosDemo.Program.LoadAccount(Int32)+0xd9
    00007F2DC13FDAB0 00007F3D7CE92B8E SosDemo.dll!SosDemo.Program.RunReport()+0x2e
HResult: 80131577

dso (dump stack objects) walks the raw stack memory and lists every object reference it finds — handy when the exception has already been caught-and-rethrown and pe on the current frame comes up empty:

> setthread 8
> dso
          SP/REG           Object Name
    7f2dc13f6478     7f2de940c9b8 System.InvalidOperationException
    7f2dc13f6808     7f2de940c890 System.Collections.Generic.KeyNotFoundException
    ...excerpt

Finalizer queue

Symptom: memory grows slowly, ~Finalize / Dispose(false) code seems to never run, or SafeHandles pile up. finalizequeue shows the state of the finalizable objects:

> finalizequeue
...
Heap 0
generation 0 has 0 objects
generation 1 has 24 objects
generation 2 has 0 objects
Ready for finalization 20,001 objects (55c0215edbb0->55c021614cb8)
------------------------------
Statistics:
          MT  Count TotalSize Class Name
...
7f4010518e20 20,000   480,000 SosDemo.PendingHandle
Total 20,025 objects, 481,872 bytes

“Ready for finalization” is the backlog: objects that are unreachable, have a finalizer, and are waiting for the finalizer thread. 20,000 stuck objects means the finalizer thread is not keeping up — or not running at all. Check it in clrthreads (it is the one marked (Finalizer)) and print its stack:

> setthread 5
> clrstack
        Child SP               IP Call Site
00007F400D11C750 ... System.Threading.Thread.Sleep(Int32)
00007F400D11C7F0 ... SosDemo.StuckFinalizer.Finalize()          [Program.cs @ 166]
00007F400D11C810 ... System.GC.RunFinalizers()

There it is: one badly-behaved finalizer blocks the single finalizer thread, so every other finalizer behind it — and everything those objects keep alive — leaks. gcroot on one of the stuck objects confirms it is held only by the queue:

> gcroot 7f307cc02048
Finalizer Queue:
    000055c0215edbb8 (finalizer root)
          -> 7f307cc02048     SosDemo.PendingHandle

Found 1 unique roots.

Runtime info

Quick orientation commands when you open an unfamiliar dump:

> eeversion
10.0.926.27113
Workstation mode
SOS Version: 10.0.14.31102

> clrmodules          # loaded managed assemblies
> dumpdomain          # AppDomain(s) and their assemblies
> eeheap -gc          # GC heap size per generation

For a deeper look at the GC heap itself (leak hunting, retention paths, objsize), see the .NET Memory Analysis with Linux post. And when you need to mix managed and native frames in the same session, that post also shows how to load SOS into LLDB.

The full list of SOS commands is in the official documentation, or type help at the dotnet dump prompt.

For my Windows 11 VM, I deliberately set the CPU type to x86-64-v4 or, alternatively, x86-64-v2-AES in the Proxmox configuration.

In combination with the latest hardware, this setting ensures that Windows runs very close to its native performance.

Another important component is the appropriate Windows VirtIO drivers. These drivers enable the virtual machine to reach its full potential. Thanks to this configuration, the Windows 11 VM is ideal for development work.

Tools such as Visual Studio run reasonably smoothly, and even larger projects can be edited without noticeable delays.

For a quick overview of system performance, simple benchmarks can be performed in Windows using the Winsat (Windows System Assessment Tool) to evaluate the performance of the VM.

Here is an example with the CPU configuration shown above and 16GB of RAM.

winsat disk

> Disk  Random 16.0 Read                       826.33 MB/s          8.6
> Disk  Sequential 64.0 Read                   3317.16 MB/s         9.3
> Disk  Sequential 64.0 Write                  3409.08 MB/s         9.3

However, I rely on Docker for software development, as many of the projects are container-based. To use Docker effectively and efficiently on Windows, the Windows Subsystem for Linux (WSL) is indispensable.

Setup WSL and Docker Desktop

Installing WSL is very simple. Open a new terminal as administrator and execute the following command

wsl --install

After successfully installing WSL, you must first restart the virtual machine so that all changes are applied correctly and the necessary system components become active. Only then can Docker Desktop be installed within the Windows VM.

After the second restart, everything seems to be ready: the virtual machine starts up cleanly, Docker Desktop opens, and the next logical step would be to start the actual container development. However, this is precisely where an unexpected problem arises.

Docker cannot be started in this configuration. The reason for this is the CPU emulation used: The configured CPU type x86-64-v4 does not provide hardware virtualization for the Windows VM.

In order for Docker to actually run within the Windows VM, an adjustment must be made in the Proxmox configuration.

The previously used CPU type x86-64-v4 must be changed to host.

After restarting, you can also check directly in Windows to see that virtualization is now available correctly. In Task Manager, under Performance, it is explicitly displayed that virtualization is active.

However, another look at the performance measurements with Winsat reveals an unpleasant finding:

The results for read and write operations are now significantly worse than before. This drop in performance is particularly noticeable in I/O-heavy workflows 🫩.

winsat disk

> Disk  Random 16.0 Read                       144.87 MB/s          7.4
> Disk  Sequential 64.0 Read                   641.51 MB/s          8.2
> Disk  Sequential 64.0 Write                  1151.43 MB/s         8.5

This is obviously bad if you have to work with Visual Studio or Rider.

Fix CPU Config

After some experimentation and help from my “AI friends” and Google, I finally came across an optimized Proxmox configuration that supports both Docker and Windows in the VM with high performance. The key settings are as follows:

args: -cpu host,svm,aes,pcid,spec-ctrl,hv_relaxed,hv_vapic,hv_vpindex,hv_time,hv_synic,hv_stimer,hv_reenlightenment,hv_frequencies
cpu: host

A new WinSAT test now shows significantly improved values 🍻

winsat disk

> Disk  Random 16.0 Read                       769.59 MB/s           8.6
> Disk  Sequential 64.0 Read                   3841.00 MB/s          9.4
> Disk  Sequential 64.0 Write                  2294.82 MB/s          9.1

To implement the optimization described above, the configuration of the corresponding VM in Proxmox must be adjusted. This can be done directly via the command line: nano /etc/pve/qemu-server/100.conf.

agent: 1
args: -cpu host,svm,aes,pcid,spec-ctrl,hv_relaxed,hv_vapic,hv_vpindex,hv_time,hv_synic,hv_stimer,hv_reenlightenment,hv_frequencies
cpu: host
balloon: 0
bios: ovmf
boot: order=scsi0;net0
cores: 20
efidisk0: local-workloads:vm-100-disk-0,efitype=4m,ms-cert=2023,pre-enrolled-keys=1,size=4M
machine: pc-q35-10.1
memory: 16000
meta: creation-qemu=10.1.2,ctime=1765131411
name: win11-test
net0: virtio=BC:24:11:29:E5:C5,bridge=vmbr0,firewall=1,tag=10
numa: 1
ostype: win11
scsi0: local-workloads:vm-100-disk-2,aio=threads,cache=writeback,iothread=1,size=50G
scsihw: virtio-scsi-single
smbios1: uuid=fefa54d2-0c4b-44ad-ac8e-79045ccd362c
sockets: 1
tpmstate0: local-workloads:vm-100-disk-1,size=4M,version=v2.0
vga: qxl
vmgenid: a81bf0f1-ad54-4df4-a7ea-6b61fff8f2ae

Further information on CPU flags and types:

• https://www.qemu.org/docs/master/system/i386/hyperv.html
• https://qemu-project.gitlab.io/qemu/system/qemu-cpu-models.html

The distrobox documentation describes the project with:

Use any Linux distribution inside your terminal. Enable both backward and forward compatibility with software and freedom to use whatever distribution you’re more comfortable with. Distrobox uses podman, docker or lilipod to create containers using the Linux distribution of your choice. The created container will be tightly integrated with the host, allowing sharing of the HOME directory of the user, external storage, external USB devices and graphical apps (X11/Wayland), and audio. https://distrobox.it/

Here is a short guide on how to create an isolated development environment for .NET 8.0 within a few minutes, including the latest JetBrains Rider 2023.3.

The development container runs with the current Debian 12 image. As host environment I use Fedora 38 or Fedora 39 Silverblue. Of course, any distribution supported by Distrobox can be used.

Create the Development Container

In a first step, we create the container for the .NET development environment with the help of distrobox create.

distrobox create -i debian -n debian --home ~/.distrobox/debian

With distrobox enter we now “jump” into the container so that all further instructions are executed in the container.

distrobox enter debian

Setup .NET 8.0

The installation of .NET 8.0 under Debian is simple. You only need the official package repositories.

wget https://packages.microsoft.com/config/debian/12/packages-microsoft-prod.deb -O packages-microsoft-prod.deb
sudo dpkg -i packages-microsoft-prod.deb

The entire .NET 8.0 SDK can now be easily installed with apt.

sudo apt update && sudo apt install dotnet-sdk-8.0

The necessary steps are explained in detail in the official Microsoft documentation.

Install JetBrains Rider 2023.3

For Rider to run smoothly, we need a few more libraries and of course git.

sudo apt install git libxtst6 libglib2.0-bin

Finally, all you need to do is download and unzip Rider.

wget https://download.jetbrains.com/rider/JetBrains.Rider-2023.3.tar.gz
tar xfvz JetBrains.Rider-2023.3.tar.gz

And you’re ready to start .NET development in a completely isolated environment 🍺

cd 'JetBrains\ Rider-2023.3/bin'
./rider.sh

Under Linux, the choice is unfortunately very limited. Fortunately, Microsoft provides a whole set of tools for analyzing memory/performance problems of .NET programs with the dotnet diagnostic tools. These work under Linux as well as in Windows and with restrictions also under macOS.

dotnet-counters

The first step is to find out if there really is a problem with the managed memory (GC Heap).

This is easily done with dotnet counters. With the help of dotnet counters monitor -p <PROCESS ID> monitoring can be activated for any .NET process.

dotnet counters monitor -p 1234

Press p to pause, r to resume, q to quit.
    Status: Running

[System.Runtime]
    % Time in GC since last GC (%)                                         0
    Allocation Rate (B / 1 sec)                                        8,168
    CPU Usage (%)                                                          0.007
    Exception Count (Count / 1 sec)                                        0
    GC Committed Bytes (MB)                                              321.516
    GC Fragmentation (%)                                                   0.046
    GC Heap Size (MB)                                                  1,074.347

Watch for GC Heap Size (MB). If the value is always increasing but never decreasing, then the application has a memory leak.

dotnet-gcdump

A simple overview of all objects managed by the garbage collector (GC) can be displayed with dotnet gcdump. To do this, first create a memory dump of the desired .NET process with dotnet gcdump collect -p <PROCESS ID>.

dotnet gcdump collect -p 1234

The managed objects can now be viewed with dotnet gcdump report <DUMP FILE>.

dotnet gcdump report 20230906_205437_23141.gcdump

  1,283,668,344  GC Heap bytes
          4,315  GC Heap objects

   Object Bytes     Count  Type
      1,048,600     1,224  System.Byte[] (Bytes > 1M)  [System.Private.CoreLib.dll]
         28,588         1  System.String (Bytes > 10K)  [System.Private.CoreLib.dll]
         16,408         1  MemoryLeak.Lib.LeakData[] (Bytes > 10K)  [MemoryLeak.Lib.dll]
         16,344         1  System.Object[] (Bytes > 10K)  [System.Private.CoreLib.dll]
          8,184         4  System.Object[] (Bytes > 1K)  [System.Private.CoreLib.dll]

The report now shows a list of all objects including their size. The output is about the same as dumpheap -stat from the SOS debugging extension.

At the moment, more information cannot be displayed with dotnet gcdump report. For memory dumps created with dotnet gcdump, the only convenient way at the moment is under Windows with the help of Perfview or Visual Studio.

dotnet-dump

dotnet-dump is the most comprehensive tool for memory analysis of .NET applications. The tool can create memory dumps of .NET processes and provides a virtual environment for analyzing these dumps.

A memory dump of any .NET process is done with dotnet dump collect -p <PROCESS ID>

dotnet dump collect -p 1234

For the analysis of the dump a virtual environment is available afterwards which is started with dotnet dump analyse <DUMP FILE> .

dotnet dump analyze core_20230830_213024

The available SOS commands are documented here and can be displayed with the help command.

An overview of all objects can be displayed with dumpheap -stat. The output is analogous to the dotnet gcdump report <DUMP FILE> used above and shows the number of all managed objects as well as their size in memory.

dumpheap -stat

Statistics:
          MT Count     TotalSize Class Name
7f230ed2e098    18           984 System.String[]
7f230ed56608     7         3,292 System.Char[]
7f230ec4b9d8    14         4,760 System.Int32[]
7f230ebda508     7        36,016 System.Object[]
7f230f1aba78     3        49,224 MemoryLeak.Lib.LeakData[]
7f230ed01038   408        49,754 System.String
7f230f1aac60 2,200        52,800 MemoryLeak.Lib.LeakData
7f230f1ab2a0 2,200        52,800 MemoryLeak.Lib.Data
56490bf28700 2,280        93,408 Free

To get more information about the object of type MemoryLeak.Lib.Data, you can use dumpheap -mt 7f230f1ab2a0 to output each instance. Here in the example there are 2’200 instances of the type MemoryLeak.Lib.Data

dumpheap -mt 7f230f1ab2a0

7ee302816f70     7f230f1ab2a0             24
7ee302816ff0     7f230f1ab2a0             24
7ee302817020     7f230f1ab2a0             24

Now that we have an overview of all objects of the MemoryLeak.Lib.Data type, we can perform further analysis. For example, we can display the “content” of the object. For this we take any object from the list and execute do (dumpobject).

do 7ee302817020

Name:        MemoryLeak.Lib.Data
MethodTable: 00007f230f1ab2a0
EEClass:     00007f230f17be00
Tracked Type: false
Size:        24(0x18) bytes
File:        /home/stef/projects/performance/MemoryLeak/bin/Debug/net7.0/MemoryLeak.Lib.dll
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
00007f230f01fcd0  400000a        8        System.Byte[]  0 instance 00007ee302e000b0 <Buffer>k__BackingField

The object contains only one byte array and is defined as follows:

internal class Data
{
    public byte[]? Buffer { get; set; }
}

Further we can display the used memory of this object with objsize:

objsize 7ee302817020

Objects which 7ee302817020(MemoryLeak.Lib.Data) transitively keep alive:

         Address               MT           Size
    7ee302817020     7f230f1ab2a0             24
    7ee302e000b0     7f230f01fcd0      1,048,600

Statistics:
          MT Count TotalSize Class Name
7f230f1ab2a0     1        24 MemoryLeak.Lib.Data
7f230f01fcd0     1 1,048,600 System.Byte[]
Total 2 objects, 1,048,624 bytes

It happens from time to time that the garbage collector does not free the memory anymore, because an object is still used somehow. Here gcroot helps. With gcroot you can see which objects reference a certain object.

gcroot 7ee302817020

HandleTable:
00007f238af413e8 (strong handle)
-> 7ee300000020 System.Object[]
-> 7ee302816f20 System.Collections.Generic.List<MemoryLeak.Lib.LeakData> (static variable: System.Random._random)
-> 7ee30201e750 MemoryLeak.Lib.LeakData[]
-> 7ee302817008 MemoryLeak.Lib.LeakData
-> 7ee302817020 MemoryLeak.Lib.Data

LLDB

For the analysis of a memory dump, LLDB can also be used instead of dotnet-dump. The advantage of LLDB is that the analysis of managed and native code is possible.

LLDB is probably available in all major linux distributions in the package sources and can be installed with the help of the package manager. Here’s an example of dnf (Fedora):

sudo dnf install lldb

The memory dump is also generated with dotnet dump collect -p <PROCESS ID>. LLDB uses the SOS debugging extension to debug managed code. It may be necessary to install dotnet-sos first so that the SOS debugging extension is found by LLDB.

dotnet tool install --global dotnet-sos
dotnet sos install

The memory dump can then be loaded into the interactive shell of LLDB with lldb –core DUMPFILE

lldb --core core_20231003_100537

Current symbol store settings:
-> Cache: /home/stef/.dotnet/symbolcache
-> Server: https://msdl.microsoft.com/download/symbols/ Timeout: 4 RetryCount: 0
(lldb) target create --core "core_20231003_100537"
Core file '/home/stef/projects/performance/dumps/core_20231003_100537' (x86_64) was loaded.
(lldb)

The memory analysis now works exactly as with dotnet dump. All SOS commands of the SOS debugging extension are available, like dumpheap -stat and so on.

More detailed information for using LLDB can be found in the official documentation from Microsoft.

dotMemory with JetBrains Rider

Of course, there are also commercial products like JetBrains Rider that are available for both Windows and Linux thanks to their platform independence. Since version 2023.2 of Rider you can collect memory dumps and analyze them directly in Rider, just like in the standalone version of dotMemory.

The problem is exacerbated in productive environments, where it is usually not possible to simply install additional software.

Fortunately, Windows also offers the possibility of recording network traffic with home remedies. The tool is called Network Shell, or netsh for short, and is able to record network traffic.

The recording can be started in an elevated command prompt as follows:

netsh trace start capture=yes tracefile=C:\capture.etl maxsize=1000 filemode=circular overwrite=yes report=no

As soon as enough data has been collected, the trace can be stopped again with the following command:

netsh trace stop

In principle, such a trace can be easily analysed with the help of the Network Monitor. Unfortunately, the Network Monitor is no longer being developed. However, the generated ETL file can also be analysed with PerfView or the Windows Performance Analyzer (WPA).

In my view, it is easier to analyse network traffic with Wireshark. Unfortunately, Wireshark cannot do much with the ETL file.

The remedy is a program published by Microsoft that is able to convert an ETL file into a PCAP file. The program, called etl2pcapng.exe, is open source and can be downloaded from GitHub under the following link:

https://github.com/microsoft/etl2pcapng

The conversion is very simple:

etl2pcapng.exe in.etl out.pcapng

And already the recorded network traffic can be analysed with the help of Wireshark 😎. This article provides further information.

This post is not primarily about what new features .NET 7.0 brings, but more about how to install the latest runtime from microsoft on fedora. Unfortunately, only .NET 6.0 can be installed via dnf. .NET 7.0 is still completely missing from the package sources.

The official documentation for .NET 7.0 and fedora only shows the not much meaningful information (Supported Version):

.NET 7 isn’t yet ready for Fedora. This article will be updated when it’s available.

whatever that means

This article is a kind of continuation of my last article about Use .NET 7 with Fedora 36 and Distrobox and describes the installation of .NET 7.0 on Fedora 37. The same procedure should most likely work for Fedora 36/35 as well.

Setup

First, head over to the official .NET 7.0 download page and select dotnet-install scripts or use wget to download the official dotnet-install.sh script.

wget https://dotnet.microsoft.com/download/dotnet/scripts/v1/dotnet-install.sh
chmod +x ./dotnet-install.sh

The installation is pretty much straightforward. Just define the specific major version with the –channel parameter to indicate a specific version. The following command installs the .NET 7.0 SDK on your computer.

./dotnet-install.sh --channel 7.0

All necessary bits and bytes are installed by default under ~/.dotnet.

ls -l ~/.dotnet
drwxr-xr-x. 1 stef stef     24 15. Nov 20:07 corefx
drwxr-xr-x. 1 stef stef      6  7. Dez 21:04 host
drwxr-xr-x. 1 stef stef      0  8. Dez 20:52 metadata
drwxr-xr-x. 1 stef stef    224  7. Dez 21:04 packs
drwxr-xr-x. 1 stef stef     28  7. Dez 21:16 sdk
drwxr-xr-x. 1 stef stef     28  7. Dez 21:07 sdk-advertising
drwxr-xr-x. 1 stef stef     52  7. Dez 21:16 sdk-manifests
drwxr-xr-x. 1 stef stef     90  7. Dez 21:04 shared
drwxr-xr-x. 1 stef stef   6300 27. Nov 19:00 symbolcache
drwxr-xr-x. 1 stef stef   3774  8. Dez 20:52 TelemetryStorageService
drwxr-xr-x. 1 stef stef     22  7. Dez 21:16 templates
drwxr-xr-x. 1 stef stef    112 27. Nov 18:55 tools
-rwxr-xr-x. 1 stef stef 175832  7. Dez 21:04 dotnet

Once the installation has been completed successfully, two environment variables must be set so that the SDK can be accessed correctly from the console. In your .bashrc set the following two environment variables:

export DOTNET_ROOT=$HOME/.dotnet
export PATH=$DOTNET_ROOT:$PATH:$DOTNET_ROOT/tools

DOTNET_ROOT variable is set to the folder .NET was installed to. And PATH should include both the DOTNET_ROOT folder and the user’s .dotnet/tools folder.

The correct installation can be checked relatively easily with the help of the dotnet command line tool:

dotnet --info

Runtime Environment:
 OS Name:     fedora
 OS Version:  37
 OS Platform: Linux
 RID:         fedora.37-x64
 Base Path:   /home/stef/.dotnet/sdk/7.0.100/

.NET SDKs installed:
  6.0.403 [/home/stef/.dotnet/sdk]
  7.0.100 [/home/stef/.dotnet/sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 6.0.11 [/home/stef/.dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 7.0.0 [/home/stef/.dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 6.0.11 [/home/stef/.dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 7.0.0 [/home/stef/.dotnet/shared/Microsoft.NETCore.App]

Environment variables:
  DOTNET_ROOT       [/home/stef/.dotnet]

JetBrains Rider

In Rider I had the problem that the SDK was not recognized. However, this can be fixed without much effort. Under File -> Settings go to Build, Execution, Deployment and select Toolset and Build.

Now change the .NET CLI executable path so that it matches your own installation.

Since this is still a preview (beta) version of .NET, I don’t want to mess up my current installation. As we all know, there are various options available for trying out new software versions or products like Docker or Virtual Machines (Gnome Boxes / VirtualBox) and so on.

In this post I want to show how to install the latest .NET Preview version on Fedora 36 Workstation without affecting the existing .NET installation. As an example I use Fedora 36 Workstation, but the steps mentioned below should also work fine under all Debian based distributions.

Setup

First, head over to the official .NET download page https://dotnet.microsoft.com/en-us/download/dotnet and select the latest preview version, which is .NET 7.0 in my case. Under SDK, select Linux and download the x64 binaries, or click this link

If you are already running .NET on your local linux system, then all the necessary dependencies should already be installed on your system. Otherwise visit https://docs.microsoft.com/en-us/dotnet/core/install/linux for further information.

Create a dedicated direcory for the preview dotnet installation and extract the downloaded package into it

mkdir -p $HOME/dotnet-preview
tar zxf dotnet-sdk-7.0.100-preview.7.22377.5-linux-x64.tar.gz -C $HOME/dotnet-preview

Define the necessary environment variables and that’s it

export DOTNET_ROOT=$HOME/dotnet-preview
export PATH=$HOME/dotnet-preview:$PATH

Check if everything is working properly with dotnet –info

dotnet --info

.NET SDK:
 Version:   7.0.100-preview.7.22377.5
 Commit:    ba310d9309

Runtime Environment:
 OS Name:     fedora
 OS Version:  36
 OS Platform: Linux
 RID:         fedora.36-x64
 Base Path:   /home/stef/dotnet-preview/sdk/7.0.100-preview.7.22377.5/

Note: We set the environment variables for the terminal session in which it was run, so when you restart the computer or even the terminal, the .NET SDK commands would no longer be found, or the “old” version would be invoked.

Distrobox

It is of course possible to store the corresponding variables in the shell profile (~/.bashrc), but this may affect the existing .NET installation. And exactly for this problem Distrobox is the perfect solution. From the official documentation:

Use any Linux distribution inside your terminal. Enable both backward and forward compatibility with software and freedom to use whatever distribution you’re more comfortable with. Distrobox uses podman or docker to create containers using the Linux distribution of your choice. The created container will be tightly integrated with the host, allowing sharing of the HOME directory of the user, external storage, external USB devices and graphical apps (X11/Wayland), and audio.

Checkout the official distrobox documentation to explore all the possibilities or watch the introduction video on Youtube by Jorge Castro.

Distrobox is included in Fedora and can be easily installed with dnf

sudo dnf install distrobox

Let’s create a new container based on Fedora 36 with a custom home directory under ~/.distrobox/dotnet-preview and named dotnet-preview

mkdir -p ~/.distrobox/dotnet-preview
distrobox create --image fedora:36 --name dotnet-preview --home ~/.distrobox/dotnet-preview

Enter into your new container with the distrobox enter command

distrobox enter dotnet-preview

After the distrobox has been started, the following commands are executed in the container you’ve just created (dotnet-preview). First, we install all updates and additional packages that are mandatory for dotnet

sudo dnf upgrade -y && dnf install nano libicu -y

The next steps are almost identical as before, with the difference that this time the dotnet package is extracted under /opt

sudo mkdir -p /opt/dotnet-preview
sudo tar zxf dotnet-sdk-7.0.100-preview.7.22377.5-linux-x64.tar.gz -C /opt/dotnet-preview

And this time, we can persist the environment variables without touching the configuration of our main system

echo 'export DOTNET_ROOT=/opt/dotnet-preview' >> ~/.bashrc
echo 'export PATH=$PATH:$DOTNET_ROOT:$DOTNET_ROOT/tools' >> ~/.bashrc

Now check your installation again with dotnet –info

dotnet --info

.NET SDK:
 Version:   7.0.100-preview.7.22377.5
 Commit:    ba310d9309

Runtime Environment:
 OS Name:     fedora
 OS Version:  36
 OS Platform: Linux
 RID:         fedora.36-x64
 Base Path:   /opt/dotnet-preview/sdk/7.0.100-preview.7.22377.5/

Now every time i want to check one of my projects for .NET 7 compatibility, i simply start my distrobox, navigate to the project and run a dotnet build or dotnet run.

The application runs as a Kubernetes Service behind a NGINX Controller. To get to the root of the problem, it is probably helpful to take a closer look at the ingress logs with the help of kubectl logs:

kubectl logs -n ingress-nginx ingress-nginx-controller-6d5f55986b-q9t7k -f

2022/02/07 19:52:42 [error] 172#172: *3825 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 192.168.49.1, server: my-app.ch, request: "GET /api/slow HTTP/1.1", upstream: "http://172.17.0.2:80/api/slow", host: "my-app.ch"
2022/02/07 19:52:43 [error] 172#172: *3825 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 192.168.49.1, server: my-app.ch, request: "GET /api/slow HTTP/1.1", upstream: "http://172.17.0.2:80/api/slow", host: "my-app.ch"
2022/02/07 19:52:44 [error] 172#172: *3825 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 192.168.49.1, server: my-app.ch, request: "GET /api/slow HTTP/1.1", upstream: "http://172.17.0.2:80/api/slow", host: "my-app.ch"
192.168.49.1 - - [07/Feb/2022:19:52:44 +0000] "GET /api/slow HTTP/1.1" 504 562 "-" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.80 Safari/537.36" 459 3.003 [myapp-myapp-80] [] 172.17.0.2:80, 172.17.0.2:80, 172.17.0.2:80 0, 0, 0 1.000, 1.002, 1.001 504, 504, 504 77e32c15c5339f90fb10c7f195f887e4

The logs quickly show where the problem lies. There is obviously a problem with the communication between the NGINX Ingress Controller and our application. It looks like the API call to http://172.17.0.2:80/api/slow runs into a timeout.

The configured timeout in nginx can be easily found out. For this we only have to output the file /etc/nginx/nginx.conf

kubectl exec -it -n ingress-nginx ingress-nginx-controller-6d5f55986b-dwnw8 -- cat /etc/nginx/nginx.conf

There are various options to configure the timeouts on the nginx. Relevant for us is the property proxy-read-timeout and this was set to 60s, which is the default.

proxy_read_timeout 60s;

Unfortunately a quick fix for the affected api was not possible, so the timeout of the ingress had to be increased. In our case, we’ve simply modified the YAML file by setting the annotation nginx.ingress.kubernetes.io/proxy-read-timeout to 180 to temporarily solve the problem.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  labels:
    app: myapp
  name: ingress-myapp
  namespace: myapp
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$1
    nginx.ingress.kubernetes.io/proxy-read-timeout: "180"
spec:
  ingressClassName: nginx
  rules:
    - host: my-app.ch
      http:
        paths:
          - path: /(.*)
            pathType: Prefix
            backend:
              service:
                name: myapp
                port:
                  number: 80

There are many reasons why the application initialization feature may fail. If the actual process is not logged, you may not even notice that the initialization was not executed as desired. The problem is that the actual application has not been started yet. Therefore the error analysis is not trivial.

Analyzing with PerfView

But you can track down the errors relatively quickly with PerfView. If you are unfamiliar with PerfView, there are PerfView video tutorials on Channel9.

To trace the application initialization of your web site, select the “Collect” item in the main menu and then choose “Collect”, or press ALT+C (click here for more details). The only thing to consider now is the use of the Microsoft-Windows-IIS ETW Provider by selecting the IIS checkbox.

Start a trace with Start Collection. Now, go to the IIS Manager and restart (start/stop) your IIS App Pool or simply press “Recycle” in the right pane under Application Pool Tasks. In Perfview you can stop the collection and double click on the Events node

The events viewer should now be displayed. It makes sense to filter the events by iis

HttpStatus 405 - Method Not Allowed

The application initialization module calls the endpoint specified under initializationPage via HTTP GET request.

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <location path="." inheritInChildApplications="false">
    <system.webServer>
      <applicationInitialization doAppInitAfterRestart="true" skipManagedModules="true">
        <add initializationPage="/api/init/start" />
      </applicationInitialization>
    </system.webServer>
  </location>
</configuration>

If the controller action does not allow a HTTP GET request, then a 405 Method Not Allowed is returned from the controller. Look in PerfView for HttpStatus 405 events like this:

Therefore, the initialization routine must be able to be called via HTTP GET request. This is usually configured with the HttpGetAttribute applied directly on the action

[ApiController]
public class InitController : ControllerBase
{
    private readonly ILogger<InitController> _logger;
    public InitController(ILogger<InitController> logger)
    {
        _logger = logger;
    }

    [HttpGet]
    [Route("api/init/start")]
    public IActionResult AppInit()
    {
        _logger.LogInformation("app initializer");
        return Ok();
    }
}

HttpStatus 401 - Unauthorized

Another common problem is the authentication of http requests. Within companies, windows authentication is often used in the intranet. however, IIS application Initialization only works with endpoints that also allow anonymous calls. For this purpose, both the web server and the application must be configured accordingly.

In the case of IIS, anonymous authentication must be enabled. In addition, the API must also allow an anonymous call. This is done with the AllowAnonymous-Attribute applied on the corresponding controller action.

[ApiController]
public class InitController : ControllerBase
{
    private readonly ILogger<InitController> _logger;
    public InitController(ILogger<InitController> logger)
    {
        _logger = logger;
    }

    [HttpGet]
    [Route("api/init/start")]
    [AllowAnonymous]
    public IActionResult AppInit()
    {
        _logger.LogInformation("app initializer");
        return Ok();
    }
}

If the controller action does not allow a Anonymous requests, then a 401 Unauthorized is returned from the controller. Look in PerfView for HttpStatus 401 events like this:

HttpStatus 301 - Web site requires SSL

Usually websites in IIS are configured with an HTTP to HTTPS rule. This means that all HTTP requests are automatically redirected to HTTPS. This is usually configured using the URL Rewrite module. An exemplary configuration of such a rule could look like this:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
    <system.webServer>
      <rewrite>
        <rules>
          <rule name="HTTP to HTTPS redirect for all requests" stopProcessing="true">
            <match url="(.*)" />
            <conditions>
              <add input="{HTTPS}" pattern="off" />
            </conditions>
            <action type="Redirect" url="https://{HTTP_HOST}/{R:1}" />
          </rule>
        </rules>
      </rewrite>
    </system.webServer>
</configuration>

Application Initialization module isn’t working for web site configured to require HTTPS only. Look in PerfView for HttpStatus 301 events like this:

The initialization endpoint must be callable with http. For this to work, the existing URL rewrite rule has to be extended to allow http calls from the app initialization module:

<rewrite>
    <rules>
        <rule name="No redirect on warmup request (request from localhost with warmup user agent)"
        stopProcessing="true">
            <match url=".*" />
            <conditions>
                <add input="{HTTP_HOST}" pattern="localhost" />
                <add input="{HTTP_USER_AGENT}" pattern="Initialization" />
            </conditions>
            <action type="Rewrite" url="{URL}" />
        </rule>
        <rule name="HTTP to HTTPS redirect for all requests" stopProcessing="true">
            <match url="(.*)" />
            <conditions>
                <add input="{HTTPS}" pattern="off" />
            </conditions>
            <action type="Redirect" url="https://{HTTP_HOST}/{R:1}" />
        </rule>
    </rules>
</rewrite>

Tip: Also read Application Initialization module fails when web site requires SSL

Conclusion

When using the IIS Application Initialization feature always ensure your web site allows anonymous calls. This means that the corresponding controller action specified in the web.config under initializationPage is marked with the AllowAnonymous attribute. Furthermore, the iis website must allow anonymous authentication.

And finally, it is always a good idea to have the IIS-HttpTracing module installed on your IIS.

The IIS 8.0 Application Initialization feature enables website administrators to configure IIS 8.0 to proactively perform initialization tasks for one or more web applications. While an application is being initialized, IIS 8.0 can also be configured to return static content as a placeholder or “splash page” until an application has completed its initialization tasks.

Before you can use Application Initialization Feature you must ensure, the appropriate IIS Module is installed on your web server. This can be done either by using the Server Manager or Powershell. In an elevated PowerShell prompt run the following command:

Enable-WindowsOptionalFeature -Online -FeatureName IIS-ApplicationInit

Tip: Use my IIS feature installation script for a complete setup of an IIS Server on Windows Server 2016/2019 and Windows 10.

ASP.NET Core Demo

Let’s start with a simple ASP.NET Core Api controller which acts as an entry point for the initialization. The idea behind this endpoint is to provide an API which will be called by the IIS Application Initialization Module to do some initialization tasks like data loading, caching etc. before the web sites handles user requests.

[ApiController]
public class InitController : ControllerBase
{
    private readonly ILogger<InitController> _logger;
    public InitController(ILogger<InitController> logger)
    {
        _logger = logger;
    }

    [HttpGet]
    [Route("api/init/start")]
    public IActionResult AppInit()
    {
        _logger.LogInformation("app initializer");
        return Ok();
    }
}

Since Application Initialization is an IIS feature, of course a web.config file is required to perform all necessary configurations. Note: /api/init/start points to my api controller route

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <location path="." inheritInChildApplications="false">
    <system.webServer>
      <applicationInitialization doAppInitAfterRestart="true" skipManagedModules="true">
        <add initializationPage="/api/init/start" />
      </applicationInitialization>
      <handlers>
        <add name="aspNetCore" path="*" verb="*" modules="AspNetCoreModuleV2" resourceType="Unspecified" />
      </handlers>
      <aspNetCore processPath="dotnet" arguments=".\dotnet-iis-preload.dll" stdoutLogEnabled="true" stdoutLogFile=".\logs\stdout" hostingModel="inprocess" />
    </system.webServer>
  </location>
</configuration>

Copy and Paste this web.config file into your Web-Project root directory and set Copy to Output Directory to always either in Visual Studio or directly inside csproj.

<Project Sdk="Microsoft.NET.Sdk.Web">
  <PropertyGroup>
    <TargetFramework>net5.0</TargetFramework>
  </PropertyGroup>
  <ItemGroup>
    <Content Update="web.config">
      <CopyToOutputDirectory>Always</CopyToOutputDirectory>
    </Content>
  </ItemGroup>
</Project>

In the configuration file shown above, i have set the property stdoutLogEnabled to true, so that the correct functionality can be checked by using the built-in logging capabilities. In productive environments, this property should always be set to false. See Log creation and redirection for further details.

Now, you’re ready to publish your web app using dotnet publish command

dotnet publish -o d:\temp\preload

Create a new IIS-Site and set the physical path to d:\temp\preload. For ASP.NET Core applications, the application pool must be set to No Managed Code.

On your site under “Advanced Settings” you must set Preload Enabled to true. With this setting, IIS sends a “fake request” to the application every time it restarts or the associated application pool starts up.

As soon your application pool has been started, you should see your app initializers log entries under .\logs\stdout

info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Production
info: Microsoft.Hosting.Lifetime[0]
      Content root path: D:\temp\preload
info: dotnet_iis_preload.Controllers.InitController[0]
      app initializer

this happens everytime your appliction pool has been started or the pool has been recycled.

The Application Initialization feature usually works quite well, but problems can occur depending on the configuration of your Web Site. In order for the feature to work properly, a few conditions must be met. See my blog post IIS application initialization - troubleshooting where I point out the possible problems and explain how to analyze and fix them.