Tuesday, June 13, 2017

SQL Cluster connection problem after removing a node

After removing a node from SQL Cluster – some machines trying to access the DB.
investigation discovered a ping timeout exactly every 5 minutes.
5 minutes is the default timeout of ARP entries.
Checking the ARP during the ping timeouts discovered the MAC address of the removed SQL node.
Shutting the node down solved the issue.
IP Conflicts…

Tuesday, June 06, 2017

Connect Secured Service Fabric Cluster by Windows Security - from non trusted domain

If you have a Service Fabric cluster which you secured using Windows security - you need to use the following powershell command to connect to it:
Connect-ServiceFabricCluster -ConnectionEndpoint "ServerName:19000" –WindowsCredential
This, however, requires you to be logged in as a memeber of the domain of the cluster or a trusted domain.
If this is not the case - you will fail to login (after long wait in my case).

If you need to connect from a different domain, you can use this workaround:
(Notice: you must run this elevated as Administrator)
runas /netonly /user:Domain\Username powershell
Give the password when asked for, and this will open a new Powershell where you can run the Connect command succesfully.

the other way is to add this Windows Credential to your Credential Manager in Control Panel:

Then you can simplly call the connect command - It will connect without any problem.

Azure Service Fabric - Windows security using gMSA - Details

On your Domain Controller:

Check if you already have a Kds Key:
Get-KdsRootKey
If not run the next line:
Add-KdsRootKey -EffectiveTime ((get-date).addhours(-10))
Validate it was created:
Get-KdsRootKey
Create the gMSA: (where its name in this sample is gMSA-SF-1, and it has 4 machines SF1,SF2,SF3,SF4)
New-ADServiceAccount -Name gMSA-SF-1 -DNSHostName gMSA-SF-1.myDomain.local -PrincipalsAllowedToRetrieveManagedPassword SF1$,SF2$,SF3$,SF4$ -ServicePrincipalNames ServiceFabric/gMSA-SF-1.myDomain.local
If later on, you need to add/remove nodes:
Set-ADServiceAccount -Identity gMSA-SF-1 -PrincipalsAllowedToRetrieveManagedPassword SF1$, SF2$, SF3$, SF4$, SF5$
Don't forget to create a Domain group with all users that should get Admin rights on the cluster using the UI - in the example below the name of this group is "SFAdmins"

On each of the cluster machines (before deploying the Service Fabric Cluster):

Add the Powershell support to manage AD:
Add-WindowsFeature RSAT-AD-PowerShell
Install the gMSA:
Install-AdServiceAccount gMSA-SF-1
Configuring the Security section:

"security": {
            "ServerCredentialType": "Windows",
            "WindowsIdentities": {
                "ClustergMSAIdentity": "mydomain.local\\gMSA-SF-1",
                "ClusterSPN": "ServiceFabric/gMSA-SF-1.mydomain.local",
                "ClientIdentities": [
                    {
                        "Identity": "mydomain.local\\SFAdmins",
                        "IsAdmin": true
                    }
                ]
            }

        },

Azure Service Fabric - Windows security on Standalone - Timeout

while trying to deploy new secured Service Fabric cluster (using Windows Security) i got the following error for all the nodes:
Timed out waiting for Installer Service to complete for machine SF1.mydomain.local. Investigation order: FabricInstallerService -> FabricSetup -> FabricDeployer -> Fabric
 When trying the same configiuration file, but without the security section, or with Certificate based security - everything worked perfectly.

Microsoft's engineer gave me a solution:
Replace your FQDN in the nodes configuration (the "iPAddress" property) to NetBios hostName.
This has solved the issue.

Wednesday, May 10, 2017

Public Cloud Only strategy - Uncompetative

I have watched Michael Dell's keynote on DELL EMC World 2017, and he said:
Anyone with a Cloud First strategy that is Public Cloud Only - I think will find himself uncompetitive for workloads that have predictable characteristics (see the video here).
Maybe this will change over time, but for now I fully agree with this.

From our experiance, even for QA environmnets that we can take down during the nights - we still pay more on the cloud than On Premise.

Monday, May 08, 2017

Crash on my next step

After seeing few human mistakes that take down a fully redundant system - I have added a new tool to my belt.

Simply tell yourself the following, before doing any configuration change:
Doing the next planned step - I need to crash the whole system - How can I make it happen?
Now you'll look for ways that a crash can really happen, instead of being sure you know it won't.

Good luck, and break a leg - not the system.

Friday, November 02, 2012

"Hurricane Sandy’s storm" or "Building Cloud based Disaster Recovery site in record time"

I was planning to write a post on how we have created a DR site from scratch in less than 3 days,
But Lahav Savir - Architect & CEO of Emind Systems (now AllCloud) - was not only working fast in creating this DR site with us, but also writing fast about this :-)

So here is a link to his report:
Disaster Recovery site Over Night, yes it happened!

Sunday, June 17, 2012

PowerShell - Memory usage per user

From time to time you might need to find out which user to logoff from a server because of low memory.

instead of looking at the Task Manager and trying to find who is using the most, you can use this PowerShell script:

$h = @{}
get-wmiobject win32_process | foreach {
    $u = $_.getowner().user;
    if ( $u -ne $null)
    {
        if ( !$h.ContainsKey($u)  )
        {
            $h.add( $u, $_.WS);
        }
        else
        {
            $h.item($u) = $h.item($u) + $_.WS;
        }
    }
} 
$h.GetEnumerator() | sort value -desc
Notice that this gives you the actual RAM usage which might not be what you are looking for, as recent users probably using more RAM than not recent ones.

If you want to know the total usage of Virtual Memory - replace "WS" with "VM".

Tuesday, July 05, 2011

ASP.NET Deadlock on WCF service hosted in IIS

One of our most important core business services is a WCF soap web service for selling our products to other companies. As we have lots of requests to those services, we must have multiple instances of this service running on several IIS servers - all behind a load balancer.

Problem begins:

Our service was running happily in production for quite a long time.
Than one day, our monitoring team noticed all instances went red in the load balancer health monitor.

EMERGENCY!!!

We immediately started a conference call with all the needed people, and as a first action we decide to recycle some instances and see if they come up ok.

They do. We were back online.

Investigation:

when we look at the event viewer, we see IIS has started to recycle some of the instances we haven’t, because of the following error:

ISAPI 'c:\windows\microsoft.net\framework\v2.0.50727\aspnet_isapi.dll' reported itself as unhealthy for the following reason: 'Deadlock detected'.

This error pointed us to this article:

Contention, poor performance, and deadlocks when you make Web service requests from ASP.NET applications

In short, the article explains that this error might be caused by the limit of the threads .Net allows to open Simultaneously, where all the allowed thread are used and they waits for something which needs thread as well.

So we have started to think if we may have such scenario in our code, and couldn’t come with this exact scenario, but we thought maybe one of our internal services was blocked and caused all the waiting thread.

Another option which we thought about was DoS attack, but we haven’t seen any increase in calls per seconds.

Anyway – we decided to follow Microsoft recommendations for the runtime configuration settings, like:

  • maxWorkerThreads
  • maxIoThreads
  • minWorkerThreads
  • minFreeThreads
  • minLocalRequestFreeThreads

And also WCF settings

  • MaxConcurrentCalls
  • MaxConcurrentInstances
  • MaxConcurrentSessions

It didn’t help – we got the same crash scenario few days later.

Investigating Dumps

We have decided to follow the instructions in “How to generate a dump file when ASP.NET deadlocks in IIS 6.0”, and created dump file the next deadlock happened.

The Dump file hasn’t shown any exception at the deadlock time, however the threads counts gave us the first clue:

  • Hundreds of ASP.NET HTTP Request threads (System.ServiceModel.Activation.HttpModule.ProcessRequest)
  • Zero WCF executing threads
    (System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin)

Logs to the help

We have collected the logs from IIS, Event Viewer, Load Balancer, Data warehouse DB, and application logs. The picture we saw was:

19:52:51 – Last successful WCF request (IIS log)
19:53:49 – 1 connection_Dropped (HTTPERR log)
19:57:45 – Last successful .aspx page request (IIS log)
20:00:24 - Deadlock Detected (W3SVC-WP Event Viewer)
20:03:35 - 164 * Connection_Abandoned_By_AppPool – most of WCF requests, few aspx pages (HTTPERR)

This gives the same picture as the Dump file:

somehow for almost 5 minutes – aspx files worked perfectly, but WCF requests were not returned to the clients (instead they were stack in the asp.net threads as there weren't any WCF threads to handle them).
All those WCF requests eat up all the available thread from the thread pool, and once finish them all (after the declared ResponseDeadlockInterval passed with no request responded) – Deadlock was declared.

Once the Process recycled – IIS abandoned all those queued requests.

What happened to the WCF?

As you can see, the dump we used was created way too late to find out the cause for the WCF problem.
But, Here logs came to the help again.

we have identified (in the IIS logs) a unique User-Agent which repeats each time just before the services crashed, so we followed this user, and using Network Monitor (Wireshark) we finally arrived to the source of the problem:

This client application has sent us requests with SOAPAction which didn’t fit to any operation in the WCF service

The Bug

We have in our WCF service an attribute - “ErrorHandledWebService” – which inherit from “Attribute” and implemets “IServiceBehavior” & “IErrorHandler” interfaces. This attribute tells WCF to bind our Error handling code the WCF flow.

When exception happens on the WCF request it triggers our “ProvideFault” method of the “IErrorHandler” interface – which we implement to write the original request to our log.

The problem starts when a request comes to the service which doesn’t fit any operation in the service an exception is thrown by WCF.

In this case - OperationContext.Current is null.

Our code used OperationContext.Current.RequestContext.RequestMessage.State which caused a Null Reference Exception which was not handled by our code.

Microsoft’s WCF Bug?

This unhandled exception inside the ProvideFault can cause one of the following:

  1. The Process terminate itself.
  2. The Process stop responding to WCF requests.

those behaviors depends on the value of this parameter:

<legacyUnhandledExceptionPolicy enabled="true/false" />

if this flag is false – the process terminate because of the unhandled exception.
if it is true – WCF is dead, but the rest functionalities (like aspx) continue to work.

as long as you the flag set to false – Microsoft behavior is just fine – crashing the process. Smile

But if the flag is true (also I know Microsoft recommend to go without the legacy mode) - I think it as a bug in the framework ,as WCF shouldn’t leave the process in an unstable state, or at least, it should write proper error message to the event log - explaining WCF is dead because of this.

What do you think?

Tuesday, March 15, 2011

service has zero application (non-infrastructure) endpoints

I got this message today after copying a working web application from one server to the other.
The web application was configured as virtual directory on both places.
Same exact web.config and same DLLs.

It turns out to be the website folder itself:

In the original working machine the website folder was valid and empty, while on the problematic server – the website was pointing to not existing folder.

I guess that when WCF looks for the web.config – it has the rule to go up to see if there are any web.config in the hierarchy above – and it failed on the not existing folder.

Hope it will help someone.

Monday, January 17, 2011

Why a good Health Check is so important in your Load Balancer

Consider the following scenario:

You have 15 Web Servers – each serves as a node of SOAP Web Service.
All services are behind a Load Balancer.

All work ok.

Suddenly your monitor shows no external request successfully served.
Ok, not all. just 99% fails…

You are checking the status of the nodes in the load balancer’s pool – All green.
You are checking 4-5 instances by calling them directly (not via the load balancer) – All good.

How could it be?

Easily:

  • The selected load balancing algorithm was “Least Connections” – which means the balancer select to redirect a coming request to the node which has the smallest number of open connections.
  • The Health check which decides whether or not an instance is “alive” was simple HTTP GET which return true for any HTTP response (as long there is a response on port 80).
  • One of the 15 instances was suffering from insufficient resources – which caused it to fail for any request it got.

Now comes the interesting part:

  • The instance which failed for each request – did it real fast…
    The other instances were responding a lot slower (they actually done some work).

The result was that the Load Balancer saw one instance which serves very fast – so almost all the requests finished being redirected to the fault instance – hence – all failed.
This was the problem.

What to do to solve the issue?

Immediate solution:

identify the fault instance (by querying all instances, one by one) – and disable it.
Automatically the Balancer return to distribute the load between all the valid nodes.

Real Solution:

Create a good Health Check.

In our case – we added an aspx file which run the business logic of the request and return “OK” as the output if it succeed. And this page and  its result was the new health check.

Conclusion:

Make sure your health check actually check the specific functionality you except to get from the nodes.
This way you promise your Load Balancer will not send requests to faulty nodes, especially not too quick ones.

Monday, December 13, 2010

iPhone Travel application - Booking Hotels Online

The company I work for has just released a new iPhone application for booking hotels online:

EZ Travel

Feel free to send me your comments, ideas etc.

Thursday, May 13, 2010

Curetracks website

CureTracks is a non-profit global collaborative project - which aims very high: finding improved pathway for curing Cancer.

The idea is smart - collect thousands samples of Cancer cure, investigate and rank them.
This way this project may be able to identify non-conventional methods which medicine has ignored.

Your help is needed!

CureTracks Website

Please share this site with your friends.
This way the project will have more cases of Cancer cure to investigate.

Friday, March 12, 2010

TFS 2008 –> TFS 2010 upgrade

 

18:00 waiting for last developers to check in their code.
18:05 R&D manager sends “Good luck”
18:15 Still waiting
18:20 Last check in
18:25 Waiting for last build to complete
18:40 Get latest sources for backup
18:45 Shutdown IIS, SharePoint and Reporting
18:45 checking no connection to DBs
18:55 Running backup task on DBs
19:27 Copying DB backup files to another machine (to be on the safe side)
19:40 still copying latest sources to another computer
20:00 Ghost backup
20:18 Detaching all DBs
20:30 Trying to boot from Win 2008 R2 installation disk.
The server doesn’t recognize the DVD.
20:40 Trying to connect USB DVD… same problem.
21:00 Trying to use Network Deployment
21:30 We get the OS task sequence on the server – but it doesn’t work
22:00 We try to fix the problem by recreating the task sequence
but now we don’t see it at all
22:05 One of us driving home to bring IDE to USB cables to connect a standard DVD Drive
22:35 He is back – it seems to work
After few minutes – the cables stop to response…
22:40 We decide to limit ourselves till 00:00  - then switch to VM
22:50 I drive home and bring my USB DVD Drive
It works!
23:00 Windows 2008 R2 doesn’t have drivers to Intel 7320 RAID controller
trying to find drivers
23:10 Searching for a floppy diskette (and a computer with floppy drive)
23:20 Drivers don’t fit
23:30 Searching…
23:55 Finding the correct drivers (64 bit)
00:00 RAID recognized!
00:03 Error 80300001 – can’t install – we are in panic
00:05 Thanks Don't Panic blog for telling us we need to return the installation media back…
00:10 Error 80070017 – burning another DVD.
00:25 First machine connected to the domain (Data Layer)
00:35 starting to Install windows updates
00:42 First restart of TSDB
00:49 Preparing To Configure Windows. Please Do Not Turn Off Your Computer…
By some blogs – this can take hours…
01:43 TSDB restarted (Only ~1 hour)
01:50 Installing .Net 3.5 on TSDB
01:53 Starting SQL 2008 installation
02:11 Installing IIS on Foundation (Application Tier)
02:16 Installing SQL 2008 SP1 on TSDB
02:17 Installing .Net 3.5 on Foundation
02:25 Installing SQL 2008 on Foundation
02:30 Problems with Permissions on drive D
03:00 Issue solved by giving the permissions again on D root cascading.
03:05 Tea break
03:18 Finish configure Reporting Service (with new DB. old one renamed)
03:30 Installing SharePoint Services and configure
04:55 After wasting 1.5 hours on permissions issue – finished to attach DBs
05:05 Installing VS 2008 on Foundation
05:10 Configure SharePoint website
05:20 Restoring old Content DB into the newly created SharePoint site
05:45 Finishing installation of VS 2008 (with all editions)
05:50 Installing VS 2010 on Foundation
06:13 Starting TFS 2010 installation
06:23 Configuration tool start to run
06:28 Configuration on step 96 (Version Control)
It should take some hours.
So we go to sleep.
10:00 Step 146…
12:40 TFS Installation completed successfully!!!
more configurations will be made later.
13:40 Our DBA crated backup task – and  ran it
13:52 Backup completed
14:46 Finished to switch folders to branches
16:27 VS 2010 installed on Build machine
16:30 Proxy server (Beta) configured to work with the upgraded instance
16:42 Team Build installed


To be continued after Shabbat…

19:20 Trying to run build
It failed because of workspace problem
Removing workspaces and try again
19:40 Build failed because of ResGen bug of long command lines.
known issue – will be solved by removing all not needed resx files.
20:50 Build still fails because workspace still connected to old URL.
Need to install compatible Team Explorer for VS 2008
21:11 Build passed!
22:30 Important build passed (after switching TF path in environment variables)
22:56 Configured our internal extension web service for TFS
23:08 Working Items customized event works
00:40 Timesheet implementation (in TFS WorkItems) works as well
01:00 Diff build doesn’t work correctly
01:11 new WIs and Categories Imported
02:30 Diff build still do make problems:
the order of the build is incorrect, and the incremental flag is ignored.
Need to see if the diff will overcome this or not
03:45 Diff still a problem – will be left for another day
04:07 Sending mail to all users

Monday, January 11, 2010

Trying to leave Skype conversation…

[23:20:10] X: Do you know you can leave a conversation? Go to -> Conversation (in the menu)--> Leave Conversation...
[23:20:17] *** X has left ***
[23:20:54] *** Y added X***
[23:20:56] Y: do you know we can add you back?
[23:21:16] Z: lol
[23:22:25] X: Do you know I can close my Skype?
[23:22:50] Y: lol

p.s. you can also write /leave in the conversation - which will cause you to leave too.

Saturday, December 05, 2009

HTTP Error 504, sc-win32-status 64, Alteon and the new employee’s phone

It started in the morning – in our QA environment.

Our topology (simplified – there are lots more connected to the LAN switch in the middle):

Network1

The QA engineer is trying to test web application (installed both on IIS server 1 & IIS Server2) via our a load balancer (Alteon in our case).

Yesterday, everything worked fine.
Today, instead of getting the homepage of the web application, we got HTTP error 504.

Quick look at the List of HTTP status codes in Wikipedia:

504 Gateway Timeout - The server was acting as a gateway or proxy and did not receive a timely request from the downstream server.

Now in English:
The Alteon, which works as a gateway between the user and the 2 web applications says it got no response from the IIS server (either 1 or 2 – or both).

a quick check of the web application – by approaching their IPs directly – showed no problem at all.
They were responsive as usual.

We have asked whether there were any network changes lately, but nobody couldn’t remember any change at all.

A deeper look at the IIS logs showed a request from the Alteon did accepted.
The HTTP response was 200 – which was strange – as the Alteon kept telling it got no response.

Then we have noticed, that also the HTTP status is 200, the sc-win32-status was 64.

We already familiar with sc-win32-status 64. but if you don’t:

The specified network name is no longer available

Which means that IIS got the request submit the response successfully to the network – only to find out it can’t reach the client who sent the request.

In our case, this error says the Alteon can see and send requests to the IIS servers, but the responses can’t reach the Alteon back.

So we changed the IP address of the Alteon Load Balancer – and – Hooray!!! it worked.

We re asked about any change in the network, this time we have specified the IP number…
And this was the real topology:

Network2

This morning a new employee join the company, and we have IP phones.
The IP Address which was given to the new employee’s IP phone was x.x.x.56…

Monday, November 16, 2009

WCF middle tier client – ClientBase proxy (svcutil) vs. ChannelFactory

After running with WCF based SOA in our production for a year or so, we have been working on our next major version. We have added some new services lately.

in the load test, we encounter the following error:

System.Net.Sockets.SocketException: Only one usage of each socket address (protocol/network address/port) is normally permitted

After some investigation (See Durgaprasad Gorti's WebLog) we understood that as we load huge amount of request on our middle tier application (actually ASP.NET web service) it send lots of requests to other services in our distributed system. We have used TCPView to see what’s going on with the ports – and saw that each request have opened a new client port – which was never reused until the OS released it 4 minutes later.

We have changed the TCPTimedWaitDelay value in the registry to 30 seconds – and it solve the problem.
we decided, however, that this is not a real solution – just a workaround. the real problem is the no reuse of ports – and we need to solve it –as we don’t know when we will hit the wall next time.

Our code used Client Proxy generated by Svcutil.exe, and we have created a new client each time a request was needed:

System.ServiceModel.ClientBase<T> clientProxy = Activator.CreateInstance(proxyType, new object[] { endpoint.Binding, endpoint.Address }) as System.ServiceModel.ClientBase<T>;

Now, one of our developers said that in .NET 3.5 (actually .NET 3.0 SP1) Microsoft has added some kind of caching which should have solved this issue.

As we were running on .NET 3.5 SP1 environment – this clearly hasn’t solved the problem. So we went to look for the reason. It hasn’t took long to find Wenlong Dong's Blog about “Performance Improvement for WCF Client Proxy Creation in .NET 3.5 and Best Practices”.

It was very clear from his blog, that we fail to use the built in caching because we use a constructor with Binding as a parameter. Actually, even if weren’t used a constructor that disable the caching – it was disabled because we accessed the ChannelFactory property too:

clientProxy.ChannelFactory.Endpoint.Behaviors.Add(epb);

We wanted to fix this by calling a different constructor and by adding the behavior to our configuration, but we couldn’t, as our configuration isn’t based on config file, but is read from a central DB – and there is no constructor that doesn’t read configuration from config files (there should be such constructor in 4.0 – See this new constructor).

You can work around this in various ways, like this one by Pablo M. Cibraro.
Read also in great details how the client actually work – at What WCF client does

Anyway, we have decide to different direction.

If you read Microsoft’s Middle-Tier Client Applications, you’ll see 2 options:

  • Cache the WCF client object and reuse it for subsequent calls where possible.
  • Create a ChannelFactory object and then use that object to create new WCF client channel objects for each call.

Clearly, the first option… is not an option, as it talks about subsequent calls.
We need robust multi threaded middle tier.

So we were left with only one valid solution – to use ChannelFactory instead of the Client proxy, and to cache it (actually redoing the caching mechanism Microsoft added for the ClientBase).
We can succeed caching in the place the framework fails, because we allow ourselves to take some assumptions regarding the reuse of the same ChannelFactory.

We use the cached factory (if not exist – we create one) to return a Typed channel by calling its CreateChannel(). This open a new socket for each request, unless already used socket were closed and available for reuse (You should of course close each created channel, by calling its Close() method).

This is what we wanted.

The load test finished successfully.

Thanks Daniel, Michael & Eyal.

Thursday, October 29, 2009

Convert folder to branch failure - Deleted folder

When you try to conevert a folder into a branch in TFS 2010, you may fail if you already have a branch somewhere in its sub folders.


The error is TF203028: You cannot create a branch at $serverPath because a branch already exists at $SubFolderServerPath.

You can fix this by converting this sub folder back to folder (as you can't have a branch inside a branch)



You might however don't see any such branch folder.
In this case - just set Visual Studio to show deleted items in Team Explorer.
Then you will be able to convert this sub folder into a folder - which will fix the problem

Bypass Gated Check in from TF Checkin Command Line (TFS 2010)

In Team System 2010, Microsoft added long wanted feature – Gated Check in.

This feature lets you promise you won’t break the build by your check in, by converting the check in into a shelveset – running a build with your changes, and only if the build succeed – it check in the code on behalf of you.

It gives you of course the option to bypass this behavior (if you have the required permission).

My problem started in the fact that during the build process itself – we perform a check in…
So this check in fail with the message (and exited with code 1):

Your check-in has been placed into shelveset Gated_XXXXXXX;Domain\User and submitted for validation by build definition \Project-Name\Build-Definition.

I have looked at MSDN for TF Checkin flag that will replace the GUI option to bypass the gated build:

msdn

But as you can see, no new option was added.

Luckily, I have run also the TF checkin /? command:

Command Line

And as you can see, there are few new options here:

  1. shelveset
  2. bypass
  3. login

Those 3 new options came to support this “Gated Check in” feature.
I was actually looking for the /bypass option – which, what a surprise, bypass this feature.

I hope that this MSDN document will be updated soon.

Monday, October 26, 2009

Upgrade from TFS 2008 to TFS 2010 Beta 2 – long wait (literally)

Yes! We have finally got the TFS 2010 Beta 2!!!

I have restored our production Team Foundation Server Databases into a clean computer.
The setup was very easy – just selecting Upgrade flow, and filling some details.

Then it arrived to the most critical part – upgrading the DBs themselves.

It took almost 6 hours…

TFS_Upgrade_Image

but… it finished successfully!

TFS_Upgrade_Image_-_success

So the proof of concept for upgrading has proofed the concept!

Now of course, we still need to see how to upgrade our customized CMMI process – to have all the new staff Microsoft put in the new one. I guess we will use Allen Clark’s Enabling New Features of Visual Studio Team System 2010 Beta 1 in Upgraded Team Projects