All of us write various automation tests for the application that we test in our every day work. One thing that has been debated the most is the use of soft assertions. Before we dive deep into the debate, a quick look at the two types of assertions.
Hard Assertions (or simply Assertions)
Tests immediately fail and stop executing the moment a failure occurs in the assertion. You may want to use a hard assert if you want to verify if you have logged in correctly and fail the test if you haven’t as there is no point in proceeding further if the test if the pre-condition itself fails.
Soft Assertions
Tests don’t stop running even if an assertion condition fails, but the test itself is marked as a failed test to indicate the right result. This is useful if you are doing multiple validations in a form (well.. you may actually question why you should be doing multiple validations in the first place which is probably a separate topic by itself), you may actually want to complete all the validations and fail the test once all validations are complete in case of failures.
Now.. there are multiple ways of doing soft asserts. You can write your own custom logic using the regular assert, but trap any exceptions and make the test proceed further. Or, you can use the TestNG library (for Java) which does this magic for you.
Here is how you initialize the TestNG assertions…
https://gist.github.com/rameshbaskar/1f0df0e227c42a94aeb5
Here is a sample test for a Hard Assertion…
https://gist.github.com/rameshbaskar/7fa17bb7b61e8e883b62
Here is a sample test for Soft Assertion which does not fail the test…
https://gist.github.com/rameshbaskar/21652650cc38eca0c8ce
And here is a soft assertion that actually marks the test as a failure…
https://gist.github.com/rameshbaskar/f938828d87f0e1cd386b
If you look at the above test, the softAssert.assertAll() does the trick. This method collates all the failures and decides whether to fail the test or not at the end. So instead of writing a custom logic, the TestNG library itself offers the facility to perform Soft Assertions in your test.
Hi,
when i use the softAssert.assertAll(); in the @AfterTest, the test is PASSED, even though my assertion is failed.
Is there a way where we can use softAssert.assertAll(); in the @AfterTest and make test Pass/Fail?
Regards,
Gajendra
public class CustomVerificationTest {
public static CustomVerification customVerification;
@Test
public void verifyTaggedBlogs() throws InterruptedException {
customVerification.verifyEquals(“this is passed test”, “s1”, “s2”);
//customVerification.checkForVerificationErrors();
}
@BeforeMethod
public void beforeTest() {
System.out.println(“BeforeTest”);
customVerification = new CustomVerification();
}
@AfterMethod
public void afterTest() {
customVerification.checkForVerificationErrors();
}
}
Hi Gajendra,
Could you please post the full CustomVerification code? I am not sure what the CustomVerification class is supposed to do in this context.
Thanks,
Ramesh
Hi Gajendra,
I have some situation that I need to manually set the testNG results for some test cases in selenium automation. could you please let me know what is the best way to do that. thanks in advance.
Hi,
@Aftertest is not the part of test cases so your test suit will show pass even @Aftertest will fail.
Thx for sharing
Would u answer why java-doc on testng.org website does not show SoftAssert type?
Its just wierd that testng.org site’s JavaDoc does not have any references to the package org.testng.asserts. However, when I generated the JavaDoc from the testng source code (v 6.8) available in my local machine, I was able to see the documentation properly.
Also the following site has the correct JavaDoc reference: http://grepcode.com/snapshot/repo1.maven.org/maven2/org.testng/testng/6.8/
Hope this helps.
Looks like they do not care properly about testng project and the project is on community hands. too sad
Thanks for the Post Ramesh….the tests now are running with out stopping when using softassert but the end report shows all as pass (green in color). My scenario is basically getting all the validation alerts in a user entry form with various data sets and comparing them with expected alerts. Following is my code snippet, Please help String actual_res = col13;
String exp_res = col12;
try{
Assert.assertEquals(exp_res, actual_res);
}catch(Throwable t){
//code to report the error in testng
ErrorUtil.addVerificationFailure(t);
System.out.println(“Response Details Message Not Matching”);
//report the error in xls file
fail=true;
}
if(fail)
{
Thread.sleep(2000);
TestUtil.reportDataSetResult(suiteTest, this.getClass().getSimpleName(), newcount+2, “FAIL”);
softAssert.assertFalse(fail,”Failed as Expected and Actual do not Match”);
}else
{
Thread.sleep(2000);
TestUtil.reportDataSetResult(suiteTest, this.getClass().getSimpleName(), newcount+2, “PASS”);
}
fail=false;
Thanks Ramesh.
The post was very useful.
However I wanted to know can we write a method or function for each failed assertion if we use the soft assertion?
If yes please provide example.
Hi Ramesh,
Thanks for the article. I am implementing the same. But I am facing other issue. If my Method1 contains some text in ‘assertall’. And after it my Method2 also fails then it shows messages from both Method1 and method2 in TestNG report. How can I make sure that msg from method2 only should show up.
A very precise and helpful post. Thanks!
Thanks ramesh for this informative post, I have tried both soft assertions and hard assertions, but In both the cases all tests are executed successfully even though assertion fails.
here is my code for reference:
@Test(priority=3)
public void ffPageElementsTestSoftAssert() {
sa.assertTrue(ffPage.ff_Type_oneway.isDisplayed());
sa.assertTrue(ffPage.ff_Type_roundtrip.isDisplayed());
sa.assertTrue(ffPage.ff_PassengerCount.isDisplayed());
sa.assertTrue(ffPage.ff_DepartingFrom.isDisplayed());
sa.assertTrue(ffPage.ff_DepartingMonth.isDisplayed());
sa.assertTrue(ffPage.ff_DepartingDay.isDisplayed());
sa.assertTrue(ffPage.ff_ArrivingIn.isDisplayed());
sa.assertTrue(ffPage.ff_ReturningMonth.isDisplayed());
sa.assertTrue(ffPage.ff_ReturningDay.isDisplayed());
sa.assertTrue(ffPage.ff_ServiceClass_EC.isDisplayed());
sa.assertTrue(ffPage.ff_ServiceClass_BC.isDisplayed());
sa.assertTrue(ffPage.ff_ServiceClass_FC.isDisplayed());
sa.assertTrue(ffPage.ff_Airline.isDisplayed());
sa.assertTrue(ffPage.ff_Continue.isDisplayed());
}
@Test(priority = 2)
public void equalityTest(){
Assert.assertEquals(5, 3);
}
@Test(priority = 1)
public void printTest1(){
Assert.assertEquals(5, 5);
}
@Test(priority = 3)
public void printTest2(){
Assert.assertEquals(“two”,”two”);
}
@Test(priority = 4)
public void printTest3(){
System.out.println(“Hello3”);
}
Output
[TestNG] Running:
C:\Users\shivkumar_tate\AppData\Local\Temp\testng-eclipse-1048462585\testng-customsuite.xml
Hello3
PASSED: printTest1
PASSED: printTest2
PASSED: printTest3
FAILED: equalityTest
java.lang.AssertionError: expected [3] but found [5]
at org.testng.Assert.fail(Assert.java:94)
at org.testng.Assert.failNotEquals(Assert.java:494)
at org.testng.Assert.assertEquals(Assert.java:123)
FAILED: ffPageElementsTestSoftAssert
java.lang.NullPointerException
at com.nt.tests.FlightFinderTest.ffPageElementsTestSoftAssert(FlightFinderTest.java:44)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.testng.internal.MethodInvocationHel
When I used soft Assertion, my testcase has verified all assert statement irrespective of whether any failed inbetween, but at the end when I use softAssert.assertAll(); In the report I see just error as ‘null’, why I don’t see what all assertions have failed with actual and expected results?
Please reply.
My code look like below :
softAssert.assertEquals(allBalancesDetails.get(balance).get(3), balance1);
softAssert.assertEquals(allBalancesDetails.get(balance).get(0), balance1ResultCode);
softAssert.assertAll();
Above, first assert is failing, and 2nd one is passing.. but at the end in result – in traces – i see null as below –
java.lang.AssertionError: The following asserts failed:
null
at org.testng.asserts.SoftAssert.assertAll(SoftAssert.java:38)
at com.sigma.rest.api.lib.Charging.balanceRequest(Charging.java:105)
at com.sigma.rest.api.test.CFBalanceRequest.cfBalanceQueryForAValidSubscriber2(CFBalanceRequest.java:38)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:84)
at org.testng.internal.Invoker.invokeMethod(Invoker.java:714)
at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:901)
at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1231)
at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
at org.testng.TestRunner.privateRun(TestRunner.java:767)
at org.testng.TestRunner.run(TestRunner.java:617)
at org.testng.SuiteRunner.runTest(SuiteRunner.java:334)
at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:329)
at org.testng.SuiteRunner.privateRun(SuiteRunner.java:291)
at org.testng.SuiteRunner.run(SuiteRunner.java:240)
at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
at org.testng.TestNG.runSuitesSequentially(TestNG.java:1224)
at org.testng.TestNG.runSuitesLocally(TestNG.java:1149)
at org.testng.TestNG.run(TestNG.java:1057)
at org.testng.remote.RemoteTestNG.run(RemoteTestNG.java:111)
at org.testng.remote.RemoteTestNG.initAndRun(RemoteTestNG.java:204)
at org.testng.remote.RemoteTestNG.main(RemoteTestNG.java:175)
softAssert.assertEquals method is taking only two parameters expected , actual. so testng does not know what error msg u want for this assertion.
So we have another method assertEquals(actual,expected,String Customised mesg)
So it can print customised mesg when this assertion fails.
SoftAssert SA = new SoftAssert();
SA.assertEquals(10,20,”Expected msg 20 but we got 10″);
when above assertion fails mentioned customised mesg is printed on console
Awesome post. I am getting the error ‘import org.testng.asserts.SoftAssert’ when I tried to import SoftAssert
The import statement that I have used in the code is ‘import org.testng.asserts.SoftAssert;’ and I am using the testNG version 5.10.0.1
I am using ANT build. Even cleaning the workspace and building again does not help.
Can some one tell me what should I do to get it running ?
SoftAssert may not be available in 5.10.0.1 Use TestNG 6.8.8
Hi, I am new in automation world, and am getting really confused about hard and soft assertion.
package revision;
import org.testng.annotations.Test;
import org.testng.asserts.Assertion;
import org.testng.asserts.SoftAssert;
public class Five {
private Assertion hardAssert = new Assertion();
private SoftAssert softAssert = new SoftAssert();
@Test
public void HardAssert() {
hardAssert.assertTrue(false);
System.out.println(“one”);
hardAssert.assertTrue(true);
System.out.println(“two”);
hardAssert.assertTrue(true);
System.out.println(“three”);
}
@Test
public void SoftAssert(){
softAssert.assertTrue(false);
softAssert.assertEquals(1,2);
softAssert.assertAll();
softAssert.assertTrue(true);
System.out.println(“five”);
softAssert.assertTrue(true);
System.out.println(“six”);
}
}
when I follow ur blog, hard assertion is working fine, but with regards to soft assertion, if i include,
softAssert.assertAll();
then the report comes as FAILED:- HardAssert() and SoftAssert()
and if I comment //softAssert.assertAll();
then the output says:- FAILED:- HardAssert() and Passed:- SoftAssert()
whereas it shouldve failed only first two lines and continued with others.
So please do help me in this
Even i faced the same issue when called AssertAll only the first testcase where assertions fails shows as failure rest of the methods get skipped or passed.I used another approach use softasserrt and capure the testcase name and exception do not use AssertAll. log them to a collection and once the exeution is done , parse the result xml file and modify the result xml with those of captured asssertions once the assettion(fail) is substitued you get the expected failures in the report . i did the trick for my cucumber automation framework
Nice post Ramesh!
This is just awesome. Thanks for help.
Is assertAll() in soft assertion considers assertion in one specific test method where it is written or it considers all the assertions in the class?
thanx
Thanks allot for information.