Archive for the ‘Nant’ Category.

Selenium browser UnitTesting from TeamCity

I just setup browser testing framework utilizing Selenium from TeamCity project. The usual suspects are involved, Gallio, MBUnit, Nant and even C# Unittests.

The usual scenario when it comes to automating browser testing is that the QA / testers will create some scripts to run a browser against your website. Somehow those tests are usually not maintained very well and often are run by hand. There is not much value in browser testing scripts if you have to run them by hand.

As I needed browser testing on one of the projects I’m on I decided to look into using more of a automated setup to run the browser tests. There seem to be two big players Waitn and Selenium, Selenium lends itself to broader range of testing, naturally we will go with Selenium.

Here is the scenario I want, the tester installs a recorder on his computer, in this case a FireFox plugin. The tester records the tests and runs them in the browser using the plugin tool. Once the tester is happy with the tests the tester checks them into the repository. After check-in tester lets a developer know that there are new or changed tests. The developer takes the script and turns it into C# UnitTest, simply has Selenium convert it to UnitTest code. Then the developer takes and updates or adds the tests that resulted from the scripts and checks it into the repository. The conversion step could be automated in the future once Selenium supports that. The next step is to run it from TeamCity and after it runs you get email with the results.

So let’s take a closer look at what is needed. We need the UnitTest to be able to run against different servers using different browsers. We will pass values from TeamCity to the Nant script that is responsible for compiling and running the tests. This is how your test C# configuration file might look like.

<!– Selenium RC properties–>
    <add key=“SeleniumAddress” value=“localhost” />
    <add key=“SeleniumPort” value=“4444” />
    <add key=“SeleniumSpeed” value=“0” />
   
    <!– Browser targets –>
    <add key=“BrowserType” value=“*firefox” />
    <add key=“BrowserUrl” value=“http://10.9.169.198/” />
    <add key=“BaseUrlPath” value=“IPCA.Dev/” />

Then the base test class will look something like this.

[FixtureSetUp]
        public virtual void TestFixtureSetup()
        {
            // Read from config
            msBrowserType = getConfigSetting(“BrowserType”, msBrowserType );
            msBrowserUrl = getConfigSetting(“BrowserUrl”, msBrowserUrl);
            msBasePath = getConfigSetting(“BaseUrlPath”, msBasePath);
            //
            msSeleniumAddress = getConfigSetting( “SeleniumAddress”, msSeleniumAddress );
            miSeleniumPort = int.Parse(getConfigSetting(“SeleniumPort”, miSeleniumPort.ToString()) );
            msSeleniumSpeed = getConfigSetting( “SeleniumSpeed”, msSeleniumSpeed );
           
           
            // Start up the selenium session, using config values
            selenium = new DefaultSelenium(msSeleniumAddress, miSeleniumPort, msBrowserType, msBrowserUrl);
            selenium.Start();
            // Clean errors
            verificationErrors = new StringBuilder();

            // sets the speed of execution of GUI commands
            selenium.SetSpeed(msSeleniumSpeed);
        }

        [TearDown]
        public void TeardownTest()
        {
            try
            {
                selenium.Stop();
            }
            catch (Exception)
            {
                // Ignore errors if unable to close the browser
            }
            Assert.AreEqual(“”, verificationErrors.ToString());
        }

And a sample Selenium C# Unittest

//
using System;
using System.Text;
using System.Text.RegularExpressions;
using System.Threading;
//
using Gallio.Framework.Assertions;
using MbUnit.Framework;
//
using Selenium;

namespace SeleniumTests
{
    [TestFixture]
    public class LoginPage : WebTestBase
    {

        [Test]
        public void TheLoginPageTest()
        {
            selenium.Open( this.msBasePath + “TestLogin.aspx”);
            selenium.Click(“lbAdmin”);
            selenium.WaitForPageToLoad(“50000”);
            selenium.Click(“loginLink”);
            selenium.WaitForPageToLoad(“50000”);
            try
            {
                Assert.IsTrue(selenium.IsTextPresent(“my responsibilities regarding permissible access”));
            }
            catch (AssertionException e)
            {
                verificationErrors.Append(e.Message);
            }
            selenium.Click(“ctl00_pageContent_btnSubmit”);
            selenium.WaitForPageToLoad(“50000”);
            try
            {
                Assert.IsTrue(selenium.IsTextPresent(“Total Unassigned Web”));
            }
            catch (AssertionException e)
            {
                verificationErrors.Append(e.Message);
            }
        }
    }
}

After the Nant script compiles the tests and is getting ready to run the UnitTests it needs to startup the Selenium engine. Make sure to spawn in order for the Selenium engine to exist on another thread than your tests.

<property name=“SeleniumExec” value=“java” />
  <property name=“SeleniumPath” value=“C:\apps\selenium\selenium-server-1.0.3\” />
  <property name=”SeleniumParams” value=”-jar ${SeleniumPath}selenium-server.jar” />

    <!– Start selenium –>
    <exec   program=”${SeleniumExec}
      commandline=”
${SeleniumParams}” workingdir=”${path.base}${WebTest}
      spawn=”
true” failonerror=”true” verbose=”true
       />

    <!– Give it a sec to load –>
    <sleep milliseconds=”3000” />

In order to run the tests using different browsers, change the configuration file of the tests before run.

<!– Run tests in Firefox browser –>
    <xmlpoke
        file=“${path.base.test}${assembly.test.config}”
        xpath=“/configuration/appSettings/add[@key=’BrowserType’]/@value”
        value=“*firefox”
        verbose=“true”/>

    <call target=“runTests” />

<target name=“runTests”
    description=“runs tests using Gallio.” >

   
    <echo message=“*** Start runTests: “/>

      <gallio
        result-property=“exitCode”
        failonerror=“false”
        report-types=“Html;Xml”
        report-directory=“${artifacts}”
        report-name-format=“gallioresults”
        show-reports=“false”
        application-base-directory=“${path.base.test}”
            >

          <!– Specify the tests assemblies  –>
          <files>
            <include name=“${path.base.test}${assembly.test}”/>
          </files>
        </gallio>

        <!–
            Set error for email injector to pick it up and GlobalFailBuildMessage for
            the end target to fail the build after cleanup
          –>

        <if test=“${int::parse(exitCode)!=0}”>
          <property name=“GlobalFailBuildMessage” value=“*** One or more tests failed. Please check the log for more details” dynamic=“true” />
          <echo message=“EmailInjectMsg=${GlobalFailBuildMessage}” />
        </if>

    <echo message=“*** End runTests: “/>
  </target>

And after the run of the Unittests Selenium needs to be shut down

<!– Stop Selenium server –>
      <get  src=“http://localhost:4444/selenium-server/driver/?cmd=shutDownSeleniumServer”
        dest=“shutdown.txt”  failonerror=“false”
      />

As I had a need to setup different configurations in TeamCity to run against different locations on the webserver I used a couple of Nant variables that are passed on the command line from TeamCity, like you normally would do when running Nant script -D:BaseUrlPath=/Test/ etc.

<echo message=“*** Location variables passed from TeamCity”/>
    <echo message=“*** BrowserUrl=${BrowserUrl} “/>
    <echo message=“*** BaseUrlPath=${BaseUrlPath} “/>

Of course you get the Gallio UnitTest report as well

gallio_report

With this setup once we deploy to a server we can run all the browser tests on it using different browsers with one click of a button from TeamCity.

Custom Nant Task

When using Nant there is not much you can say is lacking. Sometimes you find yourself wondering why you are cutting and pasting good amount of code between different scripts. There has to be a better way of reusing code and sure enough there is. Just write your own Nant extension and it’s pretty simple, let’s take a look.

First create a class that extends the nant Task then declare the name of the task and setup the properties you want available to the user. Lastly you will have ExecuteTask procedure that will be called by Nant when your Task executes. That’s all the magic !

[TaskName(“setVersionAssembly”)]
    public class CxSetVersionAssembly : NAnt.Core.Task
    {
        [TaskAttribute(“file”, Required = true)]
        [StringValidator(AllowEmpty = false)]
        public string file
        {
            get { return msAssemblyFile; }
            set { msAssemblyFile = value; }
        }

        [TaskAttribute(“buildNumber”, Required = true)]
        [StringValidator(AllowEmpty = false)]
        public string buildNumber
        {
            get { return msBuildNumber; }
            set { msBuildNumber = value; }
        }

        // Executes the nant task
        protected override void ExecuteTask()
        {
            // Don’t catch errors, it will display in the build log if any
            Project.Log(Level.Info, “start setVersionAssembly”);
            ChangeAssemblyVersionNumber(msBuildNumber, msAssemblyFile);
        }
}

When you use it in your Nant script you will have to load your assembly first. Once your assembly has been loaded you can start using your custom Task.

<loadtasks assembly=“bin\debug\DE.Nant.Extensions.dll”/>

    <setVersionAssembly
         file=“${tempFile}”
         buildNumber=“3.333.3.3000”
    />

This is a very brief and simple example, there are bunch of properties and other goodies you can use as you build out your custom Nant Tasks.

The Custom Nant Task I wrote is used to update the version of our .net assemblies during build. Attached is the complete code including Nant script and unittests. Get the code

Code coverage from Nant script using Gallio and nCover

I decided to go with MbUnit for a new project, the newest version 3.x comes bundled with Gallio. Gallio is a test runner and can run loads of different flavors of tests. nUnit, msTest, MbUnit, etc. Of course once you have your tests running you wonder how much of your code gets coverage. To figure that one out I added NCover, here is a sample of how you can have nCover cover your Gallio UnitTest runs.

For best results install Gallio on the build machine and point to that directory when you load the task
loadtasks assembly=
also make sure your list of assemblies to be covered is just the assembly name, not the file name.
assembly.list=myAssembly1;myAssembly1;etc

<!– Gallio –>
<target name=“galliounittest”
              description=“Runs MbUnit UnitTests using Gallio.” >

<echo message=“*** Start Gallio unittest: “/>

<!– Run tests –>
<loadtasks assembly=“${path.gallio.task}Gallio.NAntTasks.dll” />

<gallio
  result-property=“exitCode”
  failonerror=“false”
  runner-type=“NCover”
  report-types=“Html;Xml”
  report-directory=“${artifacts}”
  report-name-format=“gallioresults”
  show-reports=“false”
  application-base-directory=“${path.base.test}”
  >

  <runner-property value=“NCoverArguments=’//w ${path.base.test} //a ${assembly.list}'” />
  <runner-property value=“NCoverCoverageFile=’${path.ncover.dir}${coverage.xml.file}'” />
  <!– Specify the tests assemblies  –>
  <files>
    <include name=“${path.base.test}${assembly.test}”/>
  </files>
</gallio>
<fail if=“${exitCode != ‘0’}” >One or more tests failed. Please check the log for more details</fail>

<echo message=“*** End Gallio unittest: “/>
</target>

<!– NCover –>
<target name=“nCoverReport”
              description=“Creates UnitTest Coverage report.” >

<echo message=“*** Start nCoverReport: “/>

    <ncoverexplorer
      program=“${path.ncover.explorer.exe}”
      projectName=“${PojectName}”
      reportType=“ModuleClassFunctionSummary”
      outputDir=“${path.ncover.dir}”
      xmlReportName=“${coverage.xml.file}”
      htmlReportName=“${coverage.html.file}”
      showExcluded=“false”
      verbose=“True”
      satisfactoryCoverage=“1”
      failCombinedMinimum=“true”
      minimumCoverage=“0.0”>

      <fileset>
        <include name=“${path.ncover.dir}${coverage.xml.file}” />
      </fileset>
      <exclusions>
        <exclusion type=“Assembly” pattern=“*.Tests” />
        <exclusion type=“Namespace” pattern=“*.Tests*” />
      </exclusions>
    </ncoverexplorer>

<echo message=“*** End nCoverReport: “/>
</target>

NCover error

I ran into a problem when adding NCover to our UnitTests on a new project the error I got when using NCover with Gallio was the following, Profiled process terminated. Profiler connection not established. The FAQ that is in the NCover directory bundled with Gallio suggested – If using the command-line, did you COM register CoverLib.dll ?

Sure enough, just run
>regsvr32 CoverLib.dll
from a command prompt, that fixed the problem on both our build machines.

Nant, Tf Unable to determine the workspace

I was using Nant to checkout files in a build script and got the error “Unable to determine the workspace” from Team Foundation Server repository. The trick is to reset the cache first, run this command
tf workspaces /s:http://yourserver.com:8080 /noprompt
from your Nant script before doing any checkouts etc, then it should work.

Build .Net 3.5 code with Nant, MsBuild on build server without installing VS2008

If you don’t have enough space to install Visual Studio 2008 on your build machine to build .Net 3.5 solution you can do the following. Most likely you will run into the same obstacles as I did, maybe others as well.

First download Windows SDK for Windows Server 2008 and .NET Framework 3.5
http://www.microsoft.com/downloads/details.aspx?FamilyId=E6E1C3DF-A74F-4207-8586-711EBE331CDC&displaylang=en

When you install you can skip the documentation and samples, it will bring the install down from 2.4GB ! down to about 525MB, not bad.


If you get the following error when you set the framework to Net-3.5 from Nant

System.NullReferenceException: Object reference not set to an instance of an object. at NAnt.Core.FrameworkInfo.get_Version()

You have a certain version of Nant around 0.86 that has old version of the path for the SDK. In your nant.exe.config under net-3.5 look for this section

<project>
    <readregistry
        property=“installRoot”
        key=“SOFTWARE\Microsoft\.NETFramework\InstallRoot”
        hive=“LocalMachine” />

    <readregistry
        property=“sdkInstallRoot”
        key=“SOFTWARE\Microsoft\Microsoft SDKs\Windows\v6.0A\WinSDKNetFxTools\InstallationFolder”
        hive=“LocalMachine”
        failonerror=“false” />

    <property name=“frameworkDirectoryV35” value=“${path::combine(installRoot, ‘v3.5’)}” />
    <fail if=“${not(directory::exists(frameworkDirectoryV35))}”>The Framework directory for .NET 3.5 does not exist.</fail>
    <property name=“referenceV35” value=“${environment::get-folder-path(‘ProgramFiles’)}/Reference Assemblies/Microsoft/Framework/v3.5” />
    <fail if=“${not(directory::exists(referenceV35))}”>The Reference Assemblies directory for .NET 3.5 does not exist.</fail>
</project>

change the sdkInstallRoot key v6.0A value to the following
key=”SOFTWARE\Microsoft\Microsoft SDKs\Windows\v6.1\WinSDKNetFxTools\InstallationFolder”


If you get the following as well
MY.Web.csproj(364, 11): error MSB4019: The imported project “C:\Program Files\MSBuild\Microsoft\VisualStudio\v9.0\WebApplications\Microsoft.WebApplication.targets” was not found. Confirm that the path in the declaration is correct, and that the file exists on disk.

No problem, copy the content of your v9.0 directory from your dev box under C:\Program Files\MSBuild\Microsoft\VisualStudio\v9.0 to your build server, these are just configuration files.


Then if your using the Microsoft test extensions you might get this one as well
App\DataAccess\CaseDataAccessTest.cs(5, 17): error CS0234: The type or namespace name ‘VisualStudio’ does not exist in the namespace ‘Microsoft’ (are you missing an assembly reference?)

We will simply solve this one by copying the MS test dll’s from the dev machine and install them in the Gac on the build machine.
C:\tmp>gacutil /i Microsoft.VisualStudio.QualityTools.Resource.dll
C:\tmp>gacutil /i Microsoft.VisualStudio.QualityTools.UnitTestFramework.dll


Now you should be good to go and ready to build on the build server without the heavy VS2008 install.