Friday, March 12, 2010

TFS 2008 –> TFS 2010 upgrade

 

18:00 waiting for last developers to check in their code.
18:05 R&D manager sends “Good luck”
18:15 Still waiting
18:20 Last check in
18:25 Waiting for last build to complete
18:40 Get latest sources for backup
18:45 Shutdown IIS, SharePoint and Reporting
18:45 checking no connection to DBs
18:55 Running backup task on DBs
19:27 Copying DB backup files to another machine (to be on the safe side)
19:40 still copying latest sources to another computer
20:00 Ghost backup
20:18 Detaching all DBs
20:30 Trying to boot from Win 2008 R2 installation disk.
The server doesn’t recognize the DVD.
20:40 Trying to connect USB DVD… same problem.
21:00 Trying to use Network Deployment
21:30 We get the OS task sequence on the server – but it doesn’t work
22:00 We try to fix the problem by recreating the task sequence
but now we don’t see it at all
22:05 One of us driving home to bring IDE to USB cables to connect a standard DVD Drive
22:35 He is back – it seems to work
After few minutes – the cables stop to response…
22:40 We decide to limit ourselves till 00:00  - then switch to VM
22:50 I drive home and bring my USB DVD Drive
It works!
23:00 Windows 2008 R2 doesn’t have drivers to Intel 7320 RAID controller
trying to find drivers
23:10 Searching for a floppy diskette (and a computer with floppy drive)
23:20 Drivers don’t fit
23:30 Searching…
23:55 Finding the correct drivers (64 bit)
00:00 RAID recognized!
00:03 Error 80300001 – can’t install – we are in panic
00:05 Thanks Don't Panic blog for telling us we need to return the installation media back…
00:10 Error 80070017 – burning another DVD.
00:25 First machine connected to the domain (Data Layer)
00:35 starting to Install windows updates
00:42 First restart of TSDB
00:49 Preparing To Configure Windows. Please Do Not Turn Off Your Computer…
By some blogs – this can take hours…
01:43 TSDB restarted (Only ~1 hour)
01:50 Installing .Net 3.5 on TSDB
01:53 Starting SQL 2008 installation
02:11 Installing IIS on Foundation (Application Tier)
02:16 Installing SQL 2008 SP1 on TSDB
02:17 Installing .Net 3.5 on Foundation
02:25 Installing SQL 2008 on Foundation
02:30 Problems with Permissions on drive D
03:00 Issue solved by giving the permissions again on D root cascading.
03:05 Tea break
03:18 Finish configure Reporting Service (with new DB. old one renamed)
03:30 Installing SharePoint Services and configure
04:55 After wasting 1.5 hours on permissions issue – finished to attach DBs
05:05 Installing VS 2008 on Foundation
05:10 Configure SharePoint website
05:20 Restoring old Content DB into the newly created SharePoint site
05:45 Finishing installation of VS 2008 (with all editions)
05:50 Installing VS 2010 on Foundation
06:13 Starting TFS 2010 installation
06:23 Configuration tool start to run
06:28 Configuration on step 96 (Version Control)
It should take some hours.
So we go to sleep.
10:00 Step 146…
12:40 TFS Installation completed successfully!!!
more configurations will be made later.
13:40 Our DBA crated backup task – and  ran it
13:52 Backup completed
14:46 Finished to switch folders to branches
16:27 VS 2010 installed on Build machine
16:30 Proxy server (Beta) configured to work with the upgraded instance
16:42 Team Build installed


To be continued after Shabbat…

19:20 Trying to run build
It failed because of workspace problem
Removing workspaces and try again
19:40 Build failed because of ResGen bug of long command lines.
known issue – will be solved by removing all not needed resx files.
20:50 Build still fails because workspace still connected to old URL.
Need to install compatible Team Explorer for VS 2008
21:11 Build passed!
22:30 Important build passed (after switching TF path in environment variables)
22:56 Configured our internal extension web service for TFS
23:08 Working Items customized event works
00:40 Timesheet implementation (in TFS WorkItems) works as well
01:00 Diff build doesn’t work correctly
01:11 new WIs and Categories Imported
02:30 Diff build still do make problems:
the order of the build is incorrect, and the incremental flag is ignored.
Need to see if the diff will overcome this or not
03:45 Diff still a problem – will be left for another day
04:07 Sending mail to all users

Monday, January 11, 2010

Trying to leave Skype conversation…

[23:20:10] X: Do you know you can leave a conversation? Go to -> Conversation (in the menu)--> Leave Conversation...
[23:20:17] *** X has left ***
[23:20:54] *** Y added X***
[23:20:56] Y: do you know we can add you back?
[23:21:16] Z: lol
[23:22:25] X: Do you know I can close my Skype?
[23:22:50] Y: lol

p.s. you can also write /leave in the conversation - which will cause you to leave too.

Saturday, December 05, 2009

HTTP Error 504, sc-win32-status 64, Alteon and the new employee’s phone

It started in the morning – in our QA environment.

Our topology (simplified – there are lots more connected to the LAN switch in the middle):

Network1

The QA engineer is trying to test web application (installed both on IIS server 1 & IIS Server2) via our a load balancer (Alteon in our case).

Yesterday, everything worked fine.
Today, instead of getting the homepage of the web application, we got HTTP error 504.

Quick look at the List of HTTP status codes in Wikipedia:

504 Gateway Timeout - The server was acting as a gateway or proxy and did not receive a timely request from the downstream server.

Now in English:
The Alteon, which works as a gateway between the user and the 2 web applications says it got no response from the IIS server (either 1 or 2 – or both).

a quick check of the web application – by approaching their IPs directly – showed no problem at all.
They were responsive as usual.

We have asked whether there were any network changes lately, but nobody couldn’t remember any change at all.

A deeper look at the IIS logs showed a request from the Alteon did accepted.
The HTTP response was 200 – which was strange – as the Alteon kept telling it got no response.

Then we have noticed, that also the HTTP status is 200, the sc-win32-status was 64.

We already familiar with sc-win32-status 64. but if you don’t:

The specified network name is no longer available

Which means that IIS got the request submit the response successfully to the network – only to find out it can’t reach the client who sent the request.

In our case, this error says the Alteon can see and send requests to the IIS servers, but the responses can’t reach the Alteon back.

So we changed the IP address of the Alteon Load Balancer – and – Hooray!!! it worked.

We re asked about any change in the network, this time we have specified the IP number…
And this was the real topology:

Network2

This morning a new employee join the company, and we have IP phones.
The IP Address which was given to the new employee’s IP phone was x.x.x.56…

Monday, November 16, 2009

WCF middle tier client – ClientBase proxy (svcutil) vs. ChannelFactory

After running with WCF based SOA in our production for a year or so, we have been working on our next major version. We have added some new services lately.

in the load test, we encounter the following error:

System.Net.Sockets.SocketException: Only one usage of each socket address (protocol/network address/port) is normally permitted

After some investigation (See Durgaprasad Gorti's WebLog) we understood that as we load huge amount of request on our middle tier application (actually ASP.NET web service) it send lots of requests to other services in our distributed system. We have used TCPView to see what’s going on with the ports – and saw that each request have opened a new client port – which was never reused until the OS released it 4 minutes later.

We have changed the TCPTimedWaitDelay value in the registry to 30 seconds – and it solve the problem.
we decided, however, that this is not a real solution – just a workaround. the real problem is the no reuse of ports – and we need to solve it –as we don’t know when we will hit the wall next time.

Our code used Client Proxy generated by Svcutil.exe, and we have created a new client each time a request was needed:

System.ServiceModel.ClientBase<T> clientProxy = Activator.CreateInstance(proxyType, new object[] { endpoint.Binding, endpoint.Address }) as System.ServiceModel.ClientBase<T>;

Now, one of our developers said that in .NET 3.5 (actually .NET 3.0 SP1) Microsoft has added some kind of caching which should have solved this issue.

As we were running on .NET 3.5 SP1 environment – this clearly hasn’t solved the problem. So we went to look for the reason. It hasn’t took long to find Wenlong Dong's Blog about “Performance Improvement for WCF Client Proxy Creation in .NET 3.5 and Best Practices”.

It was very clear from his blog, that we fail to use the built in caching because we use a constructor with Binding as a parameter. Actually, even if weren’t used a constructor that disable the caching – it was disabled because we accessed the ChannelFactory property too:

clientProxy.ChannelFactory.Endpoint.Behaviors.Add(epb);

We wanted to fix this by calling a different constructor and by adding the behavior to our configuration, but we couldn’t, as our configuration isn’t based on config file, but is read from a central DB – and there is no constructor that doesn’t read configuration from config files (there should be such constructor in 4.0 – See this new constructor).

You can work around this in various ways, like this one by Pablo M. Cibraro.
Read also in great details how the client actually work – at What WCF client does

Anyway, we have decide to different direction.

If you read Microsoft’s Middle-Tier Client Applications, you’ll see 2 options:

  • Cache the WCF client object and reuse it for subsequent calls where possible.
  • Create a ChannelFactory object and then use that object to create new WCF client channel objects for each call.

Clearly, the first option… is not an option, as it talks about subsequent calls.
We need robust multi threaded middle tier.

So we were left with only one valid solution – to use ChannelFactory instead of the Client proxy, and to cache it (actually redoing the caching mechanism Microsoft added for the ClientBase).
We can succeed caching in the place the framework fails, because we allow ourselves to take some assumptions regarding the reuse of the same ChannelFactory.

We use the cached factory (if not exist – we create one) to return a Typed channel by calling its CreateChannel(). This open a new socket for each request, unless already used socket were closed and available for reuse (You should of course close each created channel, by calling its Close() method).

This is what we wanted.

The load test finished successfully.

Thanks Daniel, Michael & Eyal.

Thursday, October 29, 2009

Convert folder to branch failure - Deleted folder

When you try to conevert a folder into a branch in TFS 2010, you may fail if you already have a branch somewhere in its sub folders.


The error is TF203028: You cannot create a branch at $serverPath because a branch already exists at $SubFolderServerPath.

You can fix this by converting this sub folder back to folder (as you can't have a branch inside a branch)



You might however don't see any such branch folder.
In this case - just set Visual Studio to show deleted items in Team Explorer.
Then you will be able to convert this sub folder into a folder - which will fix the problem

Bypass Gated Check in from TF Checkin Command Line (TFS 2010)

In Team System 2010, Microsoft added long wanted feature – Gated Check in.

This feature lets you promise you won’t break the build by your check in, by converting the check in into a shelveset – running a build with your changes, and only if the build succeed – it check in the code on behalf of you.

It gives you of course the option to bypass this behavior (if you have the required permission).

My problem started in the fact that during the build process itself – we perform a check in…
So this check in fail with the message (and exited with code 1):

Your check-in has been placed into shelveset Gated_XXXXXXX;Domain\User and submitted for validation by build definition \Project-Name\Build-Definition.

I have looked at MSDN for TF Checkin flag that will replace the GUI option to bypass the gated build:

msdn

But as you can see, no new option was added.

Luckily, I have run also the TF checkin /? command:

Command Line

And as you can see, there are few new options here:

  1. shelveset
  2. bypass
  3. login

Those 3 new options came to support this “Gated Check in” feature.
I was actually looking for the /bypass option – which, what a surprise, bypass this feature.

I hope that this MSDN document will be updated soon.

Monday, October 26, 2009

Upgrade from TFS 2008 to TFS 2010 Beta 2 – long wait (literally)

Yes! We have finally got the TFS 2010 Beta 2!!!

I have restored our production Team Foundation Server Databases into a clean computer.
The setup was very easy – just selecting Upgrade flow, and filling some details.

Then it arrived to the most critical part – upgrading the DBs themselves.

It took almost 6 hours…

TFS_Upgrade_Image

but… it finished successfully!

TFS_Upgrade_Image_-_success

So the proof of concept for upgrading has proofed the concept!

Now of course, we still need to see how to upgrade our customized CMMI process – to have all the new staff Microsoft put in the new one. I guess we will use Allen Clark’s Enabling New Features of Visual Studio Team System 2010 Beta 1 in Upgraded Team Projects