vstest.console troubleshooting • TBlog

Intro

In previous post I’ve shown you how to fix VSTest task in Azure DevOps. This post is follow up where I share with you how to overcome problems you might face while running vstest.console.exe.

Getting the vstest.console

You have to download nuget package https://www.nuget.org/packages/Microsoft.TestPlatform/ which contains vstest.console.exe under \tools\net451\Common7\IDE\Extensions\TestPlatform.

Here is a short vstest.console documentation https://docs.microsoft.com/en-us/visualstudio/test/vstest-console-options?view=vs-2019.

Running tests

Now to execute some tests, we have to pass our test dll to vstest.console and specify logger, in this case I want Visual Studio Test Results File(trx). This is actualy the same command, that VSTest task in Azure pipeline uses.

.\vstest.console.exe C:\Data\Tests.dll /logger:trx

1	.\vstest.console.exe C:\Data\Tests.dll /logger:trx

But what if running previous snippet, we get following output:

.\vstest.console.exe : Could not find a test logger with AssemblyQualifiedName, URI or FriendlyName 'trx'.

1	.\vstest.console.exe : Could not find a test logger with AssemblyQualifiedName, URI or FriendlyName 'trx'.

Hmm, strange, lets try to remove logger argument and run again:

.\vstest.console.exe C:\Data\Tests.dll

1	.\vstest.console.exe C:\Data\Tests.dll

and now we get following output:

.\vstest.console.exe : No suitable test runtime provider found for this run.

1	.\vstest.console.exe : No suitable test runtime provider found for this run.

Well, unfortunately I don’t have much experience using vstest.console and googling No suitable test runtime provider found for this run isn’t much helpful. I’ve spent many hours trying to figure this out with no luck. But every problem has it’s solution…

Fixing

In the end I’ve realized that vstest.console has /Diag argument which, accorging to documentation: "Writes diagnostic trace logs to the specified file."

Let’s rerun our test with /Diag argument:

.\vstest.console.exe C:\Data\Tests.dll /logger:trx /Diag:C:\Data\diag.txt

1	.\vstest.console.exe C:\Data\Tests.dll /logger:trx /Diag:C:\Data\diag.txt

Investigating diagnosting log, created by previous snippet C:\Data\diag.txt, I’ve found many FileLoadExceptions, first one of them being:

TpTrace Information: 0 : 5524, 1, 2021/01/07, 10:16:34.401, 7401851795871, vstest.console.exe, AssemblyResolver.OnResolve: Microsoft.Cci: Failed to load assembly. Reason:System.IO.FileLoadException: Could not load file or assembly 'file:///C:\Program Files\Microsoft Test Platform\Common7\IDE\Extensions\TestPlatform\Extensions\Microsoft.Cci.dll' or one of its dependencies. Operation is not supported. (Exception from HRESULT: 0x80131515)
File name: 'file:///C:\Program Files\Microsoft Test Platform\Common7\IDE\Extensions\TestPlatform\Extensions\Microsoft.Cci.dll' ---> System.NotSupportedException: An attempt was made to load an assembly from a network location which would have caused the assembly to be sandboxed in previous versions of the .NET Framework. This release of the .NET Framework does not enable CAS policy by default, so this load may be dangerous. If this load is not intended to sandbox the assembly, please enable the loadFromRemoteSources switch. See http://go.microsoft.com/fwlink/?LinkId=155569 for more information.
   at System.Reflection.RuntimeAssembly._nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
   at System.Reflection.RuntimeAssembly.nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
   at System.Reflection.RuntimeAssembly.InternalLoadAssemblyName(AssemblyName assemblyRef, Evidence assemblySecurity, RuntimeAssembly reqAssembly, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
   at System.Reflection.RuntimeAssembly.InternalLoadFrom(String assemblyFile, Evidence securityEvidence, Byte[] hashValue, AssemblyHashAlgorithm hashAlgorithm, Boolean forIntrospection, Boolean suppressSecurityChecks, StackCrawlMark& stackMark)
   at System.Reflection.Assembly.LoadFrom(String assemblyFile)
   at Microsoft.VisualStudio.TestPlatform.PlatformAbstractions.PlatformAssemblyLoadContext.LoadAssemblyFromPath(String assemblyPath)
   at Microsoft.VisualStudio.TestPlatform.Common.Utilities.AssemblyResolver.OnResolve(Object sender, AssemblyResolveEventArgs args)

TpTrace Information: 0 : 5524, 1, 2021/01/07, 10:16:34.401, 7401851795871, vstest.console.exe, AssemblyResolver.OnResolve: Microsoft.Cci: Failed to load assembly. Reason:System.IO.FileLoadException: Could not load file or assembly 'file:///C:\Program Files\Microsoft Test Platform\Common7\IDE\Extensions\TestPlatform\Extensions\Microsoft.Cci.dll' or one of its dependencies. Operation is not supported. (Exception from HRESULT: 0x80131515)

File name: 'file:///C:\Program Files\Microsoft Test Platform\Common7\IDE\Extensions\TestPlatform\Extensions\Microsoft.Cci.dll' ---> System.NotSupportedException: An attempt was made to load an assembly from a network location which would have caused the assembly to be sandboxed in previous versions of the .NET Framework. This release of the .NET Framework does not enable CAS policy by default, so this load may be dangerous. If this load is not intended to sandbox the assembly, please enable the loadFromRemoteSources switch. See http://go.microsoft.com/fwlink/?LinkId=155569 for more information.

at System.Reflection.RuntimeAssembly._nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)

at System.Reflection.RuntimeAssembly.nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)

at System.Reflection.RuntimeAssembly.InternalLoadAssemblyName(AssemblyName assemblyRef, Evidence assemblySecurity, RuntimeAssembly reqAssembly, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)

at System.Reflection.RuntimeAssembly.InternalLoadFrom(String assemblyFile, Evidence securityEvidence, Byte[] hashValue, AssemblyHashAlgorithm hashAlgorithm, Boolean forIntrospection, Boolean suppressSecurityChecks, StackCrawlMark& stackMark)

at System.Reflection.Assembly.LoadFrom(String assemblyFile)

at Microsoft.VisualStudio.TestPlatform.PlatformAbstractions.PlatformAssemblyLoadContext.LoadAssemblyFromPath(String assemblyPath)

at Microsoft.VisualStudio.TestPlatform.Common.Utilities.AssemblyResolver.OnResolve(Object sender, AssemblyResolveEventArgs args)

The exception is in fact very usefull, as it says: See http://go.microsoft.com/fwlink/?LinkId=155569 for more information. In a nutshell the problem is, that vstest.console is trying to load assembly that is from a remote location and that causes previous exception.

To fix that, we have to edit vstest.console.exe.config located in same folder as vstest.console.exe(in my case C:\Program Files\Microsoft Test Platform\Common7\IDE\Extensions\TestPlatform\vstest.console.exe.config). Open vstest.console.exe.config, add <loadFromRemoteSources enabled="true"/> under <runtime> element and save. See example config bellow:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <startup useLegacyV2RuntimeActivationPolicy="true">
    <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.0" />
  </startup>
  <runtime>
    <legacyUnhandledExceptionPolicy enabled="1" />
    <!-- To get stacktrace information for portable and embedded pdbs when net472 installed on machine.
          More details https://github.com/dotnet/designs/blob/master/accepted/diagnostics/debugging-with-symbols-and-sources.md#stack-traces -->
    <AppContextSwitchOverrides value="Switch.System.Diagnostics.IgnorePortablePDBsInStackTraces=false" />
    <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
      <!-- Test adapters compiled against version 11-14, need to be redirected to version 15. -->
      <dependentAssembly>
        <assemblyIdentity name="Microsoft.VisualStudio.TestPlatform.ObjectModel" publicKeyToken="b03f5f7f11d50a3a" culture="neutral" />
        <bindingRedirect oldVersion="11.0.0.0-14.0.0.0" newVersion="15.0.0.0" />
      </dependentAssembly>
      <dependentAssembly>
        <assemblyIdentity name="Microsoft.VisualStudio.QualityTools.Tips.UnitTest.ObjectModel" publicKeyToken="b03f5f7f11d50a3a" culture="neutral" />
        <bindingRedirect oldVersion="10.0.0.0-15.0.0.0" newVersion="16.0.0.0" />
      </dependentAssembly>
    </assemblyBinding>
    <loadFromRemoteSources enabled="true"/>
  </runtime>
  .
  .
  .
</configuration>

<?xml version="1.0" encoding="utf-8"?>

</startup>

<!-- To get stacktrace information for portable and embedded pdbs when net472 installed on machine.

More details https://github.com/dotnet/designs/blob/master/accepted/diagnostics/debugging-with-symbols-and-sources.md#stack-traces -->

</dependentAssembly>

</dependentAssembly>

</assemblyBinding>

</runtime>

</configuration>

After that you should be able to successfuly run tests using vstest.console.