TeamCity

You are currently browsing the archive for the TeamCity category.

I just setup browser testing framework utilizing Selenium from TeamCity project. The usual suspects are involved, Gallio, MBUnit, Nant and even C# Unittests.

The usual scenario when it comes to automating browser testing is that the QA / testers will create some scripts to run a browser against your website. Somehow those tests are usually not maintained very well and often are run by hand. There is not much value in browser testing scripts if you have to run them by hand.

As I needed browser testing on one of the projects I’m on I decided to look into using more of a automated setup to run the browser tests. There seem to be two big players Waitn and Selenium, Selenium lends itself to broader range of testing, naturally we will go with Selenium.

Here is the scenario I want, the tester installs a recorder on his computer, in this case a FireFox plugin. The tester records the tests and runs them in the browser using the plugin tool. Once the tester is happy with the tests the tester checks them into the repository. After check-in tester lets a developer know that there are new or changed tests. The developer takes the script and turns it into C# UnitTest, simply has Selenium convert it to UnitTest code. Then the developer takes and updates or adds the tests that resulted from the scripts and checks it into the repository. The conversion step could be automated in the future once Selenium supports that. The next step is to run it from TeamCity and after it runs you get email with the results.

So let’s take a closer look at what is needed. We need the UnitTest to be able to run against different servers using different browsers. We will pass values from TeamCity to the Nant script that is responsible for compiling and running the tests. This is how your test C# configuration file might look like.

<!– Selenium RC properties–>
    <add key=“SeleniumAddress” value=“localhost” />
    <add key=“SeleniumPort” value=“4444″ />
    <add key=“SeleniumSpeed” value=“0″ />
   
    <!– Browser targets –>
    <add key=“BrowserType” value=“*firefox” />
    <add key=“BrowserUrl” value=“http://10.9.169.198/” />
    <add key=“BaseUrlPath” value=“IPCA.Dev/” />

Then the base test class will look something like this.

[FixtureSetUp]
        public virtual void TestFixtureSetup()
        {
            // Read from config
            msBrowserType = getConfigSetting(“BrowserType”, msBrowserType );
            msBrowserUrl = getConfigSetting(“BrowserUrl”, msBrowserUrl);
            msBasePath = getConfigSetting(“BaseUrlPath”, msBasePath);
            //
            msSeleniumAddress = getConfigSetting( “SeleniumAddress”, msSeleniumAddress );
            miSeleniumPort = int.Parse(getConfigSetting(“SeleniumPort”, miSeleniumPort.ToString()) );
            msSeleniumSpeed = getConfigSetting( “SeleniumSpeed”, msSeleniumSpeed );
           
           
            // Start up the selenium session, using config values
            selenium = new DefaultSelenium(msSeleniumAddress, miSeleniumPort, msBrowserType, msBrowserUrl);
            selenium.Start();
            // Clean errors
            verificationErrors = new StringBuilder();

            // sets the speed of execution of GUI commands
            selenium.SetSpeed(msSeleniumSpeed);
        }

        [TearDown]
        public void TeardownTest()
        {
            try
            {
                selenium.Stop();
            }
            catch (Exception)
            {
                // Ignore errors if unable to close the browser
            }
            Assert.AreEqual(“”, verificationErrors.ToString());
        }

And a sample Selenium C# Unittest

//
using System;
using System.Text;
using System.Text.RegularExpressions;
using System.Threading;
//
using Gallio.Framework.Assertions;
using MbUnit.Framework;
//
using Selenium;

namespace SeleniumTests
{
    [TestFixture]
    public class LoginPage : WebTestBase
    {

        [Test]
        public void TheLoginPageTest()
        {
            selenium.Open( this.msBasePath + “TestLogin.aspx”);
            selenium.Click(“lbAdmin”);
            selenium.WaitForPageToLoad(“50000″);
            selenium.Click(“loginLink”);
            selenium.WaitForPageToLoad(“50000″);
            try
            {
                Assert.IsTrue(selenium.IsTextPresent(“my responsibilities regarding permissible access”));
            }
            catch (AssertionException e)
            {
                verificationErrors.Append(e.Message);
            }
            selenium.Click(“ctl00_pageContent_btnSubmit”);
            selenium.WaitForPageToLoad(“50000″);
            try
            {
                Assert.IsTrue(selenium.IsTextPresent(“Total Unassigned Web”));
            }
            catch (AssertionException e)
            {
                verificationErrors.Append(e.Message);
            }
        }
    }
}

After the Nant script compiles the tests and is getting ready to run the UnitTests it needs to startup the Selenium engine. Make sure to spawn in order for the Selenium engine to exist on another thread than your tests.

<property name=“SeleniumExec” value=“java” />
  <property name=“SeleniumPath” value=“C:\apps\selenium\selenium-server-1.0.3\” />
  <property name=”SeleniumParams” value=”-jar ${SeleniumPath}selenium-server.jar” />

    <!– Start selenium –>
    <exec   program=”${SeleniumExec}
      commandline=”
${SeleniumParams}” workingdir=”${path.base}${WebTest}
      spawn=”
true” failonerror=”true” verbose=”true
       />

    <!– Give it a sec to load –>
    <sleep milliseconds=”3000” />

In order to run the tests using different browsers, change the configuration file of the tests before run.

<!– Run tests in Firefox browser –>
    <xmlpoke
        file=“${path.base.test}${assembly.test.config}”
        xpath=“/configuration/appSettings/add[@key='BrowserType']/@value”
        value=“*firefox”
        verbose=“true”/>

    <call target=“runTests” />

<target name=“runTests”
    description=“runs tests using Gallio.” >

   
    <echo message=“*** Start runTests: “/>

      <gallio
        result-property=“exitCode”
        failonerror=“false”
        report-types=“Html;Xml”
        report-directory=“${artifacts}”
        report-name-format=“gallioresults”
        show-reports=“false”
        application-base-directory=“${path.base.test}”
            >

          <!– Specify the tests assemblies  –>
          <files>
            <include name=“${path.base.test}${assembly.test}”/>
          </files>
        </gallio>

        <!–
            Set error for email injector to pick it up and GlobalFailBuildMessage for
            the end target to fail the build after cleanup
          –>

        <if test=“${int::parse(exitCode)!=0}”>
          <property name=“GlobalFailBuildMessage” value=“*** One or more tests failed. Please check the log for more details” dynamic=“true” />
          <echo message=“EmailInjectMsg=${GlobalFailBuildMessage}” />
        </if>

    <echo message=“*** End runTests: “/>
  </target>

And after the run of the Unittests Selenium needs to be shut down

<!– Stop Selenium server –>
      <get  src=“http://localhost:4444/selenium-server/driver/?cmd=shutDownSeleniumServer”
        dest=“shutdown.txt”  failonerror=“false”
      />

As I had a need to setup different configurations in TeamCity to run against different locations on the webserver I used a couple of Nant variables that are passed on the command line from TeamCity, like you normally would do when running Nant script -D:BaseUrlPath=/Test/ etc.

<echo message=“*** Location variables passed from TeamCity”/>
    <echo message=“*** BrowserUrl=${BrowserUrl} “/>
    <echo message=“*** BaseUrlPath=${BaseUrlPath} “/>

Of course you get the Gallio UnitTest report as well

gallio_report

With this setup once we deploy to a server we can run all the browser tests on it using different browsers with one click of a button from TeamCity.

We have the Professional version of TeamCity at work which allows 3 build agents / machines and 20 users. It’s big enough for our efforts. We just put an old build machine to bed and a new one is coming online. However I could not find how to remove agent from the TeamCity interface. First I uninstalled the agent on the old build machine, but the profile was still on the server. I ended up going into the Oracle TeamCity database and deleting the profile from the AGENTS table. After that I was able to add the new build agent in the TeamCity server console.

I decided to go with MbUnit for a new project, the newest version 3.x comes bundled with Gallio. Gallio is a test runner and can run loads of different flavors of tests. nUnit, msTest, MbUnit, etc. Of course once you have your tests running you wonder how much of your code gets coverage. To figure that one out I added NCover, here is a sample of how you can have nCover cover your Gallio UnitTest runs.

For best results install Gallio on the build machine and point to that directory when you load the task
loadtasks assembly=
also make sure your list of assemblies to be covered is just the assembly name, not the file name.
assembly.list=myAssembly1;myAssembly1;etc

<!– Gallio –>
<target name=“galliounittest”
              description=“Runs MbUnit UnitTests using Gallio.” >

<echo message=“*** Start Gallio unittest: “/>

<!– Run tests –>
<loadtasks assembly=“${path.gallio.task}Gallio.NAntTasks.dll” />

<gallio
  result-property=“exitCode”
  failonerror=“false”
  runner-type=“NCover”
  report-types=“Html;Xml”
  report-directory=“${artifacts}”
  report-name-format=“gallioresults”
  show-reports=“false”
  application-base-directory=“${path.base.test}”
  >

  <runner-property value=“NCoverArguments=’//w ${path.base.test} //a ${assembly.list}’” />
  <runner-property value=“NCoverCoverageFile=’${path.ncover.dir}${coverage.xml.file}’” />
  <!– Specify the tests assemblies  –>
  <files>
    <include name=“${path.base.test}${assembly.test}”/>
  </files>
</gallio>
<fail if=“${exitCode != ‘0′}” >One or more tests failed. Please check the log for more details</fail>

<echo message=“*** End Gallio unittest: “/>
</target>

<!– NCover –>
<target name=“nCoverReport”
              description=“Creates UnitTest Coverage report.” >

<echo message=“*** Start nCoverReport: “/>

    <ncoverexplorer
      program=“${path.ncover.explorer.exe}”
      projectName=“${PojectName}”
      reportType=“ModuleClassFunctionSummary”
      outputDir=“${path.ncover.dir}”
      xmlReportName=“${coverage.xml.file}”
      htmlReportName=“${coverage.html.file}”
      showExcluded=“false”
      verbose=“True”
      satisfactoryCoverage=“1″
      failCombinedMinimum=“true”
      minimumCoverage=“0.0″>

      <fileset>
        <include name=“${path.ncover.dir}${coverage.xml.file}” />
      </fileset>
      <exclusions>
        <exclusion type=“Assembly” pattern=“*.Tests” />
        <exclusion type=“Namespace” pattern=“*.Tests*” />
      </exclusions>
    </ncoverexplorer>

<echo message=“*** End nCoverReport: “/>
</target>

I Just updated our TeamCity environment to version 4.5.5 today, we were running 3.1 before. I had resisted an update as I always do because things just break when you update software. Since we are having some network issues it was a perfect time to take the plunge. That is after we discovered that TC 4.5+ can use Team Foundation Server repository. As the MsBuild just is not mature enough yet and there are weird quirks all around we are going to use combo TC and TFS setup for a new project we just started. Mainly running the build using nant and nunit of course.
Back to the update, I took a backup of the TC server directory just in case. Then fired up the install, the install recognized the old version and offered to get rid of it for me. The install ran fairly quickly and picked up all the configurations from the last install. After the server was up the build agents got pushed to the new version and came online in a matter of couple of minutes, now that is sweet. No manual installs on the build agent machines, its auto ! Then I just kicked off a build and everything was business as usual, it can’t get any better than this.

I came across this TeamCity Twitter plugin via StackOverflow the idea is to Tweet the build status as it changes. Pretty neat idea, however it will only run under Java 1.6. I tried to install on our build server at work but the TeamCity version we are running doesn’t run under Java 1.6 There was only one thing to do, write my own that will work under Java 1.5. It has to do authenticated POST request to the Twitter REST API. The build Twitter account can be found here, not that your that interested in our build Twitter feed :)

//
import java.io.*;
import java.net.Authenticator;
import java.net.PasswordAuthentication;
import java.net.URL;
import java.net.URLEncoder;
import java.net.URLConnection;
//
import jetbrains.buildServer.serverSide.BuildServerAdapter;
import jetbrains.buildServer.serverSide.SBuildServer;
import jetbrains.buildServer.serverSide.SRunningBuild;
import jetbrains.buildServer.messages.Status;
//
import com.intellij.openapi.diagnostic.Logger;

// Tweets build start / finish to the build account
public class TwitterNotifier extends BuildServerAdapter
{
   private final static Logger LOG = Logger.getInstance(TwitterNotifier.class.getName());
   
   
   // Constructor, register the listener ( this )
   public TwitterNotifier(SBuildServer aBuildServer)
   {
       LOG.info(“*** Notifier adding listener=” + this );

       // Register with TC
       aBuildServer.addListener(this);
   }

   // Build started, set the version number
   public void buildStarted(SRunningBuild build)
   {
       String sTcBuildNumber = build.getBuildNumber();
       sendTweet( “Build “ + sTcBuildNumber + ” started on “ + build.getAgentName() );
   }

   public void buildChangedStatus(SRunningBuild build, Status oldStatus, Status newStatus)
   {
           String sTcBuildNumber = build.getBuildNumber();
       sendTweet( “Build “ + sTcBuildNumber + ” status, “ + newStatus.getText() );
   }
     
   public void buildInterrupted(SRunningBuild build)
   {
           String sTcBuildNumber = build.getBuildNumber();
       sendTweet( “Build “ + sTcBuildNumber + ” interrupted status, “ + build.getBuildStatus().getText() );
   }
   
   public void buildFinished(SRunningBuild build)
   {
           String sTcBuildNumber = build.getBuildNumber();
       sendTweet( “Build “ + sTcBuildNumber + ” finished, “ + build.getBuildStatus().getText() );          
   }
   
   // Updates the twitter account
   public static void sendTweet( String psText )
   {
           try
           {
               LOG.info( “*** TwitterNotifier plugin, “ + psText );
              
               //  Using paramaters known to work
               String urlString = “http://twitter.com/statuses/update.xml”;
               String data = “status=” + URLEncoder.encode( psText, “UTF-8″);
               String username = “twitteruser”;
               String password = “twitterpassword”;
              
               // 
               Authenticator.setDefault(new MyAuthenticator(username, password));
               // Send data
                URL url = new URL(  urlString );
                URLConnection conn = url.openConnection();
                conn.setDoOutput(true);
                OutputStreamWriter out = new OutputStreamWriter(conn.getOutputStream());
                out.write(data);
                out.flush();
                
                // Get the response
                StringBuffer sb = new StringBuffer();
                BufferedReader readIn = new BufferedReader(new InputStreamReader(conn.getInputStream()));
                String line;
                while ((line = readIn.readLine()) != null)
                        sb.append(line + \n );

                //
                //System.out.println( "*** Response=" + sb.toString() );
                
                // Close       
                out.close();
                readIn.close();
           }
           catch( Exception ex )
           {
               LOG.info( “***ERROR=” + exception2string( ex ) );
           }
   }

   public static String exception2string(Exception e)
   {
      try
      {
        StringWriter sw = new StringWriter();
        PrintWriter pw = new PrintWriter(sw);
        e.printStackTrace(pw);
        return “——\r\n + sw.toString() + “——\r\n;
      }
      catch(Exception ex)
      {
        return “error converting exception to string”;
      }
   }

   
   static class MyAuthenticator extends Authenticator
   {
       private String username, password;

       public MyAuthenticator(String user, String pass)
       {
         username = user;
         password = pass;
       }

       protected PasswordAuthentication getPasswordAuthentication()
       {
         /*
         System.out.println("Requesting Host  : " + getRequestingHost());
         System.out.println("Requesting Port  : " + getRequestingPort());
         System.out.println("Requesting Prompt : " + getRequestingPrompt());
         System.out.println("Requesting Protocol: "
             + getRequestingProtocol());
         System.out.println("Requesting Scheme : " + getRequestingScheme());
         System.out.println("Requesting Site  : " + getRequestingSite());
         */

         return new PasswordAuthentication(username, password.toCharArray());
       }
   }

}  // EOC
 

TeamCity gets thumbs up !

A fellow developer and I implemented TeamCity at work.  Before we had elaborate Nant build script, but the end result was limited.  We only got a fail / success email at the end of the build.  If you get a fail you will have to search for the failure, was it that the database didn’t build or the code ?  Not only are you bound to fail on bad check-ins but also on external factors.  Like the network drive not mounted, out of space or the local drive of the build machine itself.  It could even be an update to the build script which in many cases is harder to detect.  It could also be an UnitTest because of change in code.  Whoever is monitoring the build has to take the time to figure these things out right away and if you don’t they will snowball.

Enter TeamCity which will in most cases pinpoint the failures for you, in a easy to navigate web interface.  Not only that but there are endless useful features in TeamCity.  Such as how long each of your tests took to run with history back in time.  Before you run a build you can see the pending changes and you can even diff the changes to the files in the browser, again this history is kept through time .  Before with a single running nant script we would spit out to a text file the build success / failures to have a audit log.  To open up the audit log and figure out which build succeeded and what changes were in there, uhm just wasn’t that clean, besides not everybody remembers to tag the audit log with the relevant info.  However  with TeamCity all of that is right there on the main page, you just navigate through the configuration you want to see and all the stats are there.  Failure / success, how long it took to run, how long it took to run on the different build boxes, how many UnitTest were run etc.

Besides from that TeamCity offers pre-tested commit.  The way it works is that if your code compiles clean and the build succeeds your code will be checked into the repository.  If not you will get a failure and your code will not be checked in and your fellow developers won’t be affected by the offending bad checkins that you would have had normally in the repository.  We all know how frustrating it is when somebody checks in their code and leaves for the day just to break the build and leave everybody in a non compile state if they dared to get the latest code from the repository.  That leads to a lot of wasted time, we have all done it at some point, some more than others.  At work we are getting to the point were we can start testing the pre-tested commits, we are pretty excited about that.

TeamCity is one of those product were you take a look and you just like it at first sight.  Then you start looking deeper and you find out it’s pretty solid.  It’s true that I would have offered a couple of things out of the box, but they actually built it so it can be extended via plugins, which is always a bonus.  If  you can’t find the feature your looking for odds are that you can write a little plugin to do the task.  TeamCity works with all major repos and databases to keep it’s data.

For a full feature set you should visit TeamCity for the whole story and guess what, if you are in a small to mid size development shop it’s actually FREE.  Check the professional edition On the other hand if your a bigger shop you need to shell out some $$ it’s definetely worth it.

« Older entries