Skip to content
Steve Gilham edited this page Jan 1, 2023 · 52 revisions

I'm using dotnet Nuke (or some other build script solution) rather than Fake or Cake - how do I use AltCover?

AltCover has its own extensive APIs, requiring only implementations of the various ...Options interfaces, such as are provided in the AltCover.DotNet assembly, to drive the tool directly, in process, through dotnet test or as a separate process.

For driving dotnet test to collect coverage, then as part of the response to the regression in the dotnet 7.0.100 SDK, APIs were also added that reveal the plain property/value pairs for the dotnet test command line, rather than the pre-composed "/p:property=value" strings. Simply follow the Cake example with the locally implemented ...Options types to set up the coverage parameters, then invoke the AltCover.DotNet.ToTestPropertiesList as per the 7.0.100 workround to get the pairs to be passed into the Nuke dotnet test wrapper.

To drive the command line tool out of process, the WhatIf extension methods transform the ...Options values to the appropriate command line, with ahead-of-time checks.

I'm using dotnet SDK v7.0.100 and everything's broken. Help!!

Update to v7.0.101 or later. Failing that see this page for issues with the first net7.0 SDK release, and how to work around using versions up to 8.4.840; and this page for working around the issue with the extended features in the 8.5.841 release.

What's up with the incomplete branch coverage on this switch statement?

Except in cases where the branches are a set of adjacent integers, so the IL switch instruction can be used, C# switch statements (and F# match expressions) compile to a (possibly complex and nested) series of if/else branches. In its implementation, AltCover follows Opencover in, by default, showing a warts-and-all version of coverage : your program has some logic in it that is not being tested, even if (as is usually the case) they all turn out to be different and obscure paths to the default case; or in some cases, are not even coverable.

For those wanting a more reassuring set of results for their reports, the --visibleBranches option applies a heuristic to omit those branches that look like they are compiler-added paths from a switch-type construct. As a coverage tool has to process code from a broad range of sources, this is not a true decompilation, however, and is of course vulnerable to compiler changes which may create both false positive and negative cases.

See issue 72 for background and issue 171 for another instance.

How do I exclude an assembly from coverage?

All the various filters are lists of regexes. In the case of assemblies, the regex matches against the assembly name, so for My.Assembly.dll the string of interest is (typically) just My.Assembly; the regex can be aimed at any suitably distinctive substring, noting that . is a wildcard in this context -- so to exclude all My.[whatever] assemblies but not the assembly in Myopia.dll, use ^My\. as the filter.

How do I install the AltCover CLI?

Well, there's the dotnet altcover tool as either local or global, via

dotnet tool install [--local|--global] altcover.global --version [version]

and Visualizer

dotnet tool install [--local|--global] altcover.visualizer --version [version]

Otherwise, from the base package, just as any other NuGet packaged tool e.g. OpenCover or ReportGenerator; so for the default NuGet install location

$(UserProfile)\.nuget\packages\altcover\[version]\tools\net472\AltCover.exe

or

dotnet $(UserProfile)\.nuget\packages\altcover\[version]\tools\netcoreapp2.0\AltCover.dll

and the Visualizer for Framework/Mono is at

$(UserProfile)\.nuget\packages\altcover\[version]\tools\net472\AltCover.Visualizer.exe

With the altcover.api package the CLI tool paths are like altcover.api\[version]\lib rather than altcover\[version]\tools (N.B. there is no visualizer in this package)

How does AltCover handle this problem during CI?

For third-party dotnet tools, through the initial dotnet tool restore; for traditional third-party tools like OpenCover, by having a source-free and non-built project containing a ProjectReference to the NuGet package, which is then subject to dotnet restore --packages in the set-up phase, with some XML work to determine the path to each tool from the corresponding ProjectReference tag; for self-testing (to validate the packaging process), by getting the NuGet package and using System.IO.Compression.ZipFile.ExtractToDirectory

How do I get my F# inline functions to show whether they are partly or fully covered?

Unfortunately, the only way is to do the coverage runs with the inline declaration conditionally commented out. The inlined code is just added to the call site, with no debug reference to the original source location. Nor is there any indication in the IL that a function was declared inline, so it can't be optionally suppressed in the way that e.g. automatic properties are. See also https://github.com/dotnet/fsharp/issues/9176

Note that should F# issue 9176 be resolved simply by matching code from the inlined block back to the original source location, as happens with C++/CLI (needs OpenCover to actually take gather the data -- see below and https://github.com/SteveGilham/altcover/issues/77), the inlined function itself will still show as not covered in the XML, as there is no link from the copy to the original. Report generating programs that aggregate at the source level may display some coverage information, depending how the multiple instances of the same source location from different methods are combined.

If I do dotnet test my.sln how do I get separate coverage files for each test project?

As of release v7.1.795, if you include the literal $(ProjectName) in the report file path, then that will be explicitly replaced, in the circumstances that MSBuild hasn't done so already, so, for example:

/p:AltCoverReport=/_temp/$(ProjectName).coverage.xml

will drop a set of distinctly named files into the temporary folder.

This also works for the tokens $(SolutionDir) and $([System.Guid]::NewGuid())

How do I tell which test covered which line, like with OpenCover's coverbytest option?

The --callContext command line option (or equivalently named parameter for other means of invocation) does this, and in a more general fashion.

To track all unit tests, use the attribute name in square brackets option, giving your unit test framework's attribute, like [Test], [Fact], [TestMethod], or whatever.
To track a subset of tests, either mark them with a custom attribute, or give an explicit list of test method names (either by multiple --callContext options for the command line, a | separated list for MSBuild or dotnet test, or as an appropriately typed collection of names for everything else)
For integration test cases, where the focus is what happens per API call, track the API methods either by name or a common attribute
For integration test cases, where test steps can be marked against wall clock time, e.g. finding what is covered by a user operation that causes a sequence of API calls, use the timer interval option to match visits with test steps.

  • Method and attribute name matches are exact (apart from the optional Attribute part of an attribute type name), not wild-card or regex based.
  • Branch visits are also tracked

I only have .net core 3 or above on my machine -- how do I get things to work?

Upgrade to v6.5 or later. From v5.3, the global tools should work anyway.

The command-line tool AltCover.dll can also be run using the roll forwards option as in dotnet --roll-forward Major AltCover.dll ..., as should the old-style dotnet-cli tool as in dotnet --roll-forward Major altcover ... If required, the blunt instrument is to set the environment variable DOTNET_ROLL_FORWARD to have value Major.

If I use the Quick Start, I get the coverage including the unit test code or a third-party test runner. How do I exclude the unit test related code like this?

Use the -e or -assemblyExcludeFilter command line argument (the /p:AltCoverAssemblyExcludeFilter MSBuild parameter). to exclude unit test assemblies (or other assemblies in the test stack which depend on the ones you're interested in covering e.g. test adapters). This stops any visit instrumentation being written, but will still perform any necessary rewriting to link correctly against the instrumented assemblies, and to apply --callContext tracking.

My system under test drags in a large number of other assemblies, so I'm having to use a whole bunch of filter exclusions. Is there a way to just opt-in an assembly to instrument?

Yes; from release 6.2.714 filter expressions with a leading ? are treated as excluding non-matching, so "exclude all strings that don't have substring 'xyzzy'" can be written as "?xyzzy". With earlier releases, more complicated and fragile expressions using negative lookahead were required like "^((?!xyzzy).)*$", which had problems scaling.

My system under test invokes some platform specific (possibly native code) libraries that appear in x86 or x64 subdirectories, but these aren't carried over into the instrumented code, so running with the instrumented test assembly, the test set fails for not finding them. How do I fix this?

From version 8.2.820, this should no longer be an issue, as the input sub-directories are also copied. For earlier builds, the --inplace option is probably what you need here; then you can point the unit tests at the instrumented assembly in $(TargetDir). Alternatively, you need a manual copy step between instrumenting and testing.

What about using AltCover on C++/CLI assemblies?

The Mono.Cecil library used to instrument the assembles does not read the non-IL code in mixed-mode assemblies, and does not write assemblies without the ILOnly core flag set. As the C++/CLI compiler does not reliably set that flag even using the now deprecated /clr:pure compiler option, that effectively excludes that language from support.

OpenCover does provide coverage data for the managed parts of C++/CLI assemblies on Windows, while OpenCppCoverage will provide coverage for pure native .dll/.exe files; this currently leaves a gap in support for the native parts of mixed-mode assemblies. I do have a PR in to fix this omission, which you could build yourself while waiting for any activity on that repo.

The global tool visualizer doesn't seem to work. What gives?

tl;dr version -- Upgrade to the latest version (7.1.780 or later).

For the previous implementation, having GTK+3 installed on your machine and in your PATH was a pre-req. For Windows, this requires jumping through a few hoops, see e.g. https://github.com/GtkSharp/GtkSharp/wiki/Installing-Gtk-on-Windows

Also, GTK+ needs the GSETTINGS_SCHEMA_DIR environment variable set to pick up standard assets.

If that variable isn't set by your install, then you can make it a part of the process-level environment, by command-line argument --schemadir=path-of-schema-directory -- that value will be saved into the application configuration for later uses. For GTK+ installed as above on a vanilla 64-bit system, the appropriate path is C:\msys64\mingw64\share\glib-2.0\schemas

I do "Instrument now, test later" style coverage gathering, and I get zero CRAP scores.

Yes, coverage percentages, and any quantities derived from those percentages, are not computed in the end-of-process handler. Use a "runner" mode to perform those calculations in their own process, or use the Write-OpenCoverDerivedState cmdlet (or the underlying Toolkit API AltCover.OpenCover.PostProcess) which extracts just that step of the "runner" operations.

Clone this wiki locally