Specflow Test Execution Report Enhancement
“Specflow”….. you lost my Tags from the feature file !!!
This post should help you get back your tags from the feature file into the HTML Specflow Report.
But before that, let us take a quick look at what is Specflow ?
“Specflow”, to simply speak, is a tool that can help you automate your test cases aka scenarios by writing them in Gherkin language(don’t worry it’s more English and not like C, C++, java, C# etc) in a file viz “Feature” which your Business Analysts / Customers/ Domain Experts can understand. This same feature file will be hooked to the underlying implementations by your .Net developers. Thereby narrowing down the gap between the domain experts and your .Net developers. For more details on Specflow, I recommend reading its quick introduction page at http://www.specflow.org/getting-started/
With that quick background let’s start with today’s topic i.e. enhancement to the existing Specflow Test Execution Report (STER).
Below is my feature file where I am testing a calculator for addition of 2 numbers. Please note the tags
@RequirementNumber-1 and @RequirementNumber-2
Sample Feature File
Now, Let’s run the test using Ms Test and generate a Test Results(.trx) file. The command to generate the html STER from a Ms Test Results file can be found @ http://www.specflow.org/documentation/Reporting/. The STER looks as below for our calculator
Specflow Default Test Execution Report
Notice that the tags we added in the feature file are not shown in the HTML report, this happens because of 2 reasons
- When internally Specflow convert the Ms Test Report to a n-unit test report (xml) through an internal xslt file “MSTestToNUnit.xslt”, the tags/test category node is excluded.
- The default internal xslt file “NUnitExecutionReport.xslt” for converting the n-unit test report to HTML file does not include styling instructions for the nunit-categories node.
Now, the documentation of STER states that we can pass our own customized style sheet as a parameter to the Specflow.exe command line interface, however while generating the report it is observed that the customized xslt is applied to the n-unit xml file and not to the original Ms Test results file therefore if any information such as “TestCategory” is lost in the first transformation step itself then it will never appear in the final html STER.
To address this issue and if you want to run your test through Ms Test with Test Categories included in the report then you can do the following
1. Download the Specflow code from the github repository
2. Update the MSTestToNUnit.xslt file with the one that I have uploaded here
3. Compile the Specflow solution or update the TechTalk.SpecFlow.Reporting.dll in your reference folders.
Running the Specflow command through command prompt with the same Ms Test Results (.trx) file will now generate a node for “TestCategory” from the trx file in the n-unit test xml as follows.
NUnit Report XML
This enables us to include the Test Category aka Tags in our HTML report as shown in the snapshot below with a customized xslt (download link) thus making the Automated Test execution report complete.
Specflow Test Execution Report_Enhanced
Hope that helps and makes each user story & its acceptance criteria traceable to its automated test scenario.
DISCLAIMER : The views, opinions and information compiled in this blog are completely mine and has nothing to do with any of my previous, current or future employers
Really nice, do you have also a xslt for the Nunitreport?
Thnx Michael, Please download the xslt by clicking here
The Specflow team has merged my changes into their product, therefore the latest release should be having the MSTest changes. The NUnit xslt was not submitted and is now available only on my blog post(as of now).
Regards
Rakesh
Thanks a lot, I tried it but every scenario were uncollapsed including the failure,.. Do you have an idea?
Hi Michael,
Is it possible for you to share the generated html report and the trx file ? and let me know the specflow version you are using.
Regards
Rakesh
Can you mailed me? I am using specflow 1.9.0 But there is no need for trx file for an NUnit Report, if am right?
My bad, I assumed you are using mstest. I have emailed you the xslt file and its dependent files from source code. It should work.
Hi,
I replace MSTestToNUnit.xslt with your file
and then copy TechTalk.SpecFlow.Reporting.dll to my project
but the report remain unchanged
Do you know why?
Hi reut,
Below is a text from one of the emails that I recently sent to one.of my readers. I will come up with another detailed post on this but can you try the following or share your email id and i will send u the zip file. Also for changing the report you will need to change or download the nunit xlst file, the link for which is in one of my last comments.
<<
Some points which I excluded from my post as it focused mainly on MsTest.
The NUnitExecutionReport.xslt internally calls 3 more files which are available in the sourcecode of specflow.
I would suggest to maintain the folder structure
The folder structure is as follows
+ Reporting (Folder)
++Common (Folder)
+++Common.xslt
+++ GherkinElements.xslt
+++Languages.xml
+NUnitExecutionReport (Folder)
++NUnitExecutionReport.xslt
>>>
Thanks for the fast comment 🙂
Can you send me the zip to my email please:
[email protected]
Thanks!
Done..check your mailbox, hope it works
Hi,
excuse my ignorance, I am fairly new to VS application, but how do I “update the TechTalk.SpecFlow.Reporting.dll in my reference folders”….
I followed your instructions and compiled the Specflow solution downloaded from GitHub and copied the “TechTalk.SpecFlow.Reporting.dll’ from the Specflow solution downloaded from GitHub via->C:\Repo\SpecFlow-2\packages\SpecFlow.1.9.0\tools directory of the specflow solution to ..\packages\SpecFlow.1.9.0\tools directory of my solution. However, my HTML report is now no longer being generated. Please advise. My email address is [email protected] thanks
Hi Troy,
I have replied you over the email.
Since there was no new release of nuget package from Specflow team, we have to do the following high level steps if and only if you are using mstest as your unit test provider
1. Download latest Specflow code from github
2. Compile it (no manual changes needed as the required changes already exists in the code).
3. Use it for generating reports.
For customization of HTML report(mstest/nunit), I’ve already attached the main xslt file in my post. There are dependent xslt files which you can get from the source code of specflow.
Hope you were able to get your report working.
Yes, I got it working in the end. Thanks… #GreatBlog ..thanks for sharing.
Hi I was wondering how you created NUnit Report XML (as in your blog above). As there is some scenario output in the test result. When I use nunit(-console) output is only created separately.
Hi Danny,
I’ve generated a trx file through command prompt using mstest.
Once the trx file is generated, execute the following command giving reference to the generated trx file
specflow.exe mstestexecutionreport BookShop.AcceptanceTests.csproj /testResult:result.trx /out:MyResult.xml
Note : Instead of html in the out parameter, I have specified xml and that’s when the xml output was generated.
If you are using Nunit-console, you need not take this path as the tags are not missed. The specflow tags are available and can be made visible on Html output if you pass a custom XSLT like I did.
Hope the above explanation helps.
I receive an Xslt compile error when using the nunitexecutionreport.xslt file from this blog post. I added the /xsltFile argument to the command line statement with a value of the downloaded file. Has anyone else ran into this issue?
Hi George,
The xslt file requires other supporting xslt and xml files. These are in the source code of specflow but I have tried collecting them in one folder at https://github.com/softwarecookie/SpecflowSamples/tree/master/SpecflowCustomReporting/Reporting.
Try copying the reporting folder. The github location is a work in progress code.
Hope it helps. Let me know how it goes
Thanks, that worked. However I am running into an issue now where I have a scenario outline scenario. When I apply the your xslt template and run to insert the extra column into the report, it works for a single scenarios, but inserts a ‘-‘ for every scenario outline.
I was able to figure out how to insert the tags for scenario outlines. When nunit builds the xml file for parameterized tests, it doesn’t add a category node under each scenario in the xml file. In order to add the tags for each row in the html output we neeed to tweak the template to look for parameterized tests and use those tags. See code below.
–
Thanks George.. thats awesome 🙂
Welcome, for some reason the code snippet didn’t post, prob due to the XML tags. I’ll make an update and create a pull request in github later today