Tuesday, June 13, 2017
SQL Cluster connection problem after removing a node
investigation discovered a ping timeout exactly every 5 minutes.
5 minutes is the default timeout of ARP entries.
Checking the ARP during the ping timeouts discovered the MAC address of the removed SQL node.
Shutting the node down solved the issue.
IP Conflicts…
Tuesday, June 06, 2017
Connect Secured Service Fabric Cluster by Windows Security - from non trusted domain
Connect-ServiceFabricCluster -ConnectionEndpoint "ServerName:19000" –WindowsCredentialThis, however, requires you to be logged in as a memeber of the domain of the cluster or a trusted domain.
If this is not the case - you will fail to login (after long wait in my case).
If you need to connect from a different domain, you can use this workaround:
(Notice: you must run this elevated as Administrator)
runas /netonly /user:Domain\Username powershellGive the password when asked for, and this will open a new Powershell where you can run the Connect command succesfully.
the other way is to add this Windows Credential to your Credential Manager in Control Panel:
Then you can simplly call the connect command - It will connect without any problem.
Azure Service Fabric - Windows security using gMSA - Details
Check if you already have a Kds Key:
Get-KdsRootKey
Add-KdsRootKey -EffectiveTime ((get-date).addhours(-10))
Get-KdsRootKey
New-ADServiceAccount -Name gMSA-SF-1 -DNSHostName gMSA-SF-1.myDomain.local -PrincipalsAllowedToRetrieveManagedPassword SF1$,SF2$,SF3$,SF4$ -ServicePrincipalNames ServiceFabric/gMSA-SF-1.myDomain.local
Don't forget to create a Domain group with all users that should get Admin rights on the cluster using the UI - in the example below the name of this group is "SFAdmins"Set-ADServiceAccount -Identity gMSA-SF-1 -PrincipalsAllowedToRetrieveManagedPassword SF1$, SF2$, SF3$, SF4$, SF5$
Add-WindowsFeature RSAT-AD-PowerShellInstall the gMSA:
Install-AdServiceAccount gMSA-SF-1
Azure Service Fabric - Windows security on Standalone - Timeout
When trying the same configiuration file, but without the security section, or with Certificate based security - everything worked perfectly.Timed out waiting for Installer Service to complete for machine SF1.mydomain.local. Investigation order: FabricInstallerService -> FabricSetup -> FabricDeployer -> Fabric
Microsoft's engineer gave me a solution:
Replace your FQDN in the nodes configuration (the "iPAddress" property) to NetBios hostName.This has solved the issue.
Wednesday, May 10, 2017
Public Cloud Only strategy - Uncompetative
Anyone with a Cloud First strategy that is Public Cloud Only - I think will find himself uncompetitive for workloads that have predictable characteristics (see the video here).Maybe this will change over time, but for now I fully agree with this.
From our experiance, even for QA environmnets that we can take down during the nights - we still pay more on the cloud than On Premise.
Monday, May 08, 2017
Crash on my next step
Doing the next planned step - I need to crash the whole system - How can I make it happen?Now you'll look for ways that a crash can really happen, instead of being sure you know it won't.
Good luck, and break a leg - not the system.
Friday, November 02, 2012
"Hurricane Sandy’s storm" or "Building Cloud based Disaster Recovery site in record time"
But Lahav Savir - Architect & CEO of Emind Systems (now AllCloud) - was not only working fast in creating this DR site with us, but also writing fast about this :-)
So here is a link to his report:
Disaster Recovery site Over Night, yes it happened!
Sunday, June 17, 2012
PowerShell - Memory usage per user
instead of looking at the Task Manager and trying to find who is using the most, you can use this PowerShell script:
$h = @{}Notice that this gives you the actual RAM usage which might not be what you are looking for, as recent users probably using more RAM than not recent ones.
get-wmiobject win32_process | foreach {
$u = $_.getowner().user;
if ( $u -ne $null)
{
if ( !$h.ContainsKey($u) )
{
$h.add( $u, $_.WS);
}
else
{
$h.item($u) = $h.item($u) + $_.WS;
}
}
}
$h.GetEnumerator() | sort value -desc
If you want to know the total usage of Virtual Memory - replace "WS" with "VM".
Tuesday, July 05, 2011
ASP.NET Deadlock on WCF service hosted in IIS
One of our most important core business services is a WCF soap web service for selling our products to other companies. As we have lots of requests to those services, we must have multiple instances of this service running on several IIS servers - all behind a load balancer.
Problem begins:
Our service was running happily in production for quite a long time.
Than one day, our monitoring team noticed all instances went red in the load balancer health monitor.
EMERGENCY!!!
We immediately started a conference call with all the needed people, and as a first action we decide to recycle some instances and see if they come up ok.
They do. We were back online.
Investigation:
when we look at the event viewer, we see IIS has started to recycle some of the instances we haven’t, because of the following error:
ISAPI 'c:\windows\microsoft.net\framework\v2.0.50727\aspnet_isapi.dll' reported itself as unhealthy for the following reason: 'Deadlock detected'.
This error pointed us to this article:
In short, the article explains that this error might be caused by the limit of the threads .Net allows to open Simultaneously, where all the allowed thread are used and they waits for something which needs thread as well.
So we have started to think if we may have such scenario in our code, and couldn’t come with this exact scenario, but we thought maybe one of our internal services was blocked and caused all the waiting thread.
Another option which we thought about was DoS attack, but we haven’t seen any increase in calls per seconds.
Anyway – we decided to follow Microsoft recommendations for the runtime configuration settings, like:
- maxWorkerThreads
- maxIoThreads
- minWorkerThreads
- minFreeThreads
- minLocalRequestFreeThreads
And also WCF settings
- MaxConcurrentCalls
- MaxConcurrentInstances
- MaxConcurrentSessions
It didn’t help – we got the same crash scenario few days later.
Investigating Dumps
We have decided to follow the instructions in “How to generate a dump file when ASP.NET deadlocks in IIS 6.0”, and created dump file the next deadlock happened.
The Dump file hasn’t shown any exception at the deadlock time, however the threads counts gave us the first clue:
- Hundreds of ASP.NET HTTP Request threads (System.ServiceModel.Activation.HttpModule.ProcessRequest)
- Zero WCF executing threads
(System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin)
Logs to the help
We have collected the logs from IIS, Event Viewer, Load Balancer, Data warehouse DB, and application logs. The picture we saw was:
19:52:51 – Last successful WCF request (IIS log)
19:53:49 – 1 connection_Dropped (HTTPERR log)
19:57:45 – Last successful .aspx page request (IIS log)
20:00:24 - Deadlock Detected (W3SVC-WP Event Viewer)
20:03:35 - 164 * Connection_Abandoned_By_AppPool – most of WCF requests, few aspx pages (HTTPERR)
This gives the same picture as the Dump file:
somehow for almost 5 minutes – aspx files worked perfectly, but WCF requests were not returned to the clients (instead they were stack in the asp.net threads as there weren't any WCF threads to handle them).
All those WCF requests eat up all the available thread from the thread pool, and once finish them all (after the declared ResponseDeadlockInterval passed with no request responded) – Deadlock was declared.
Once the Process recycled – IIS abandoned all those queued requests.
What happened to the WCF?
As you can see, the dump we used was created way too late to find out the cause for the WCF problem.
But, Here logs came to the help again.
we have identified (in the IIS logs) a unique User-Agent which repeats each time just before the services crashed, so we followed this user, and using Network Monitor (Wireshark) we finally arrived to the source of the problem:
This client application has sent us requests with SOAPAction which didn’t fit to any operation in the WCF service
The Bug
We have in our WCF service an attribute - “ErrorHandledWebService” – which inherit from “Attribute” and implemets “IServiceBehavior” & “IErrorHandler” interfaces. This attribute tells WCF to bind our Error handling code the WCF flow.
When exception happens on the WCF request it triggers our “ProvideFault” method of the “IErrorHandler” interface – which we implement to write the original request to our log.
The problem starts when a request comes to the service which doesn’t fit any operation in the service an exception is thrown by WCF.
In this case - OperationContext.Current is null.
Our code used OperationContext.Current.RequestContext.RequestMessage.State which caused a Null Reference Exception which was not handled by our code.
Microsoft’s WCF Bug?
This unhandled exception inside the ProvideFault can cause one of the following:
- The Process terminate itself.
- The Process stop responding to WCF requests.
those behaviors depends on the value of this parameter:
<legacyUnhandledExceptionPolicy enabled="true/false" />
if this flag is false – the process terminate because of the unhandled exception.
if it is true – WCF is dead, but the rest functionalities (like aspx) continue to work.
as long as you the flag set to false – Microsoft behavior is just fine – crashing the process.
But if the flag is true (also I know Microsoft recommend to go without the legacy mode) - I think it as a bug in the framework ,as WCF shouldn’t leave the process in an unstable state, or at least, it should write proper error message to the event log - explaining WCF is dead because of this.
What do you think?
Tuesday, March 15, 2011
service has zero application (non-infrastructure) endpoints
I got this message today after copying a working web application from one server to the other.
The web application was configured as virtual directory on both places.
Same exact web.config and same DLLs.
It turns out to be the website folder itself:
In the original working machine the website folder was valid and empty, while on the problematic server – the website was pointing to not existing folder.
I guess that when WCF looks for the web.config – it has the rule to go up to see if there are any web.config in the hierarchy above – and it failed on the not existing folder.
Hope it will help someone.
Monday, January 17, 2011
Why a good Health Check is so important in your Load Balancer
Consider the following scenario:
You have 15 Web Servers – each serves as a node of SOAP Web Service.
All services are behind a Load Balancer.
All work ok.
Suddenly your monitor shows no external request successfully served.
Ok, not all. just 99% fails…
You are checking the status of the nodes in the load balancer’s pool – All green.
You are checking 4-5 instances by calling them directly (not via the load balancer) – All good.
How could it be?
Easily:
- The selected load balancing algorithm was “Least Connections” – which means the balancer select to redirect a coming request to the node which has the smallest number of open connections.
- The Health check which decides whether or not an instance is “alive” was simple HTTP GET which return true for any HTTP response (as long there is a response on port 80).
- One of the 15 instances was suffering from insufficient resources – which caused it to fail for any request it got.
Now comes the interesting part:
- The instance which failed for each request – did it real fast…
The other instances were responding a lot slower (they actually done some work).
The result was that the Load Balancer saw one instance which serves very fast – so almost all the requests finished being redirected to the fault instance – hence – all failed.
This was the problem.
What to do to solve the issue?
Immediate solution:
identify the fault instance (by querying all instances, one by one) – and disable it.
Automatically the Balancer return to distribute the load between all the valid nodes.
Real Solution:
Create a good Health Check.
In our case – we added an aspx file which run the business logic of the request and return “OK” as the output if it succeed. And this page and its result was the new health check.
Conclusion:
Make sure your health check actually check the specific functionality you except to get from the nodes.
This way you promise your Load Balancer will not send requests to faulty nodes, especially not too quick ones.
Monday, December 13, 2010
iPhone Travel application - Booking Hotels Online
EZ Travel
Feel free to send me your comments, ideas etc.
Thursday, May 13, 2010
Curetracks website
The idea is smart - collect thousands samples of Cancer cure, investigate and rank them.
This way this project may be able to identify non-conventional methods which medicine has ignored.
Your help is needed!
CureTracks Website
Please share this site with your friends.
This way the project will have more cases of Cancer cure to investigate.
Friday, March 12, 2010
TFS 2008 –> TFS 2010 upgrade
| 18:00 | waiting for last developers to check in their code. |
| 18:05 | R&D manager sends “Good luck” |
| 18:15 | Still waiting |
| 18:20 | Last check in |
| 18:25 | Waiting for last build to complete |
| 18:40 | Get latest sources for backup |
| 18:45 | Shutdown IIS, SharePoint and Reporting |
| 18:45 | checking no connection to DBs |
| 18:55 | Running backup task on DBs |
| 19:27 | Copying DB backup files to another machine (to be on the safe side) |
| 19:40 | still copying latest sources to another computer |
| 20:00 | Ghost backup |
| 20:18 | Detaching all DBs |
| 20:30 | Trying to boot from Win 2008 R2 installation disk. The server doesn’t recognize the DVD. |
| 20:40 | Trying to connect USB DVD… same problem. |
| 21:00 | Trying to use Network Deployment |
| 21:30 | We get the OS task sequence on the server – but it doesn’t work |
| 22:00 | We try to fix the problem by recreating the task sequence but now we don’t see it at all |
| 22:05 | One of us driving home to bring IDE to USB cables to connect a standard DVD Drive |
| 22:35 | He is back – it seems to work After few minutes – the cables stop to response… |
| 22:40 | We decide to limit ourselves till 00:00 - then switch to VM |
| 22:50 | I drive home and bring my USB DVD Drive It works! |
| 23:00 | Windows 2008 R2 doesn’t have drivers to Intel 7320 RAID controller trying to find drivers |
| 23:10 | Searching for a floppy diskette (and a computer with floppy drive) |
| 23:20 | Drivers don’t fit |
| 23:30 | Searching… |
| 23:55 | Finding the correct drivers (64 bit) |
| 00:00 | RAID recognized! |
| 00:03 | Error 80300001 – can’t install – we are in panic |
| 00:05 | Thanks Don't Panic blog for telling us we need to return the installation media back… |
| 00:10 | Error 80070017 – burning another DVD. |
| 00:25 | First machine connected to the domain (Data Layer) |
| 00:35 | starting to Install windows updates |
| 00:42 | First restart of TSDB |
| 00:49 | Preparing To Configure Windows. Please Do Not Turn Off Your Computer… By some blogs – this can take hours… |
| 01:43 | TSDB restarted (Only ~1 hour) |
| 01:50 | Installing .Net 3.5 on TSDB |
| 01:53 | Starting SQL 2008 installation |
| 02:11 | Installing IIS on Foundation (Application Tier) |
| 02:16 | Installing SQL 2008 SP1 on TSDB |
| 02:17 | Installing .Net 3.5 on Foundation |
| 02:25 | Installing SQL 2008 on Foundation |
| 02:30 | Problems with Permissions on drive D |
| 03:00 | Issue solved by giving the permissions again on D root cascading. |
| 03:05 | Tea break |
| 03:18 | Finish configure Reporting Service (with new DB. old one renamed) |
| 03:30 | Installing SharePoint Services and configure |
| 04:55 | After wasting 1.5 hours on permissions issue – finished to attach DBs |
| 05:05 | Installing VS 2008 on Foundation |
| 05:10 | Configure SharePoint website |
| 05:20 | Restoring old Content DB into the newly created SharePoint site |
| 05:45 | Finishing installation of VS 2008 (with all editions) |
| 05:50 | Installing VS 2010 on Foundation |
| 06:13 | Starting TFS 2010 installation |
| 06:23 | Configuration tool start to run |
| 06:28 | Configuration on step 96 (Version Control) It should take some hours. So we go to sleep. |
| 10:00 | Step 146… |
| 12:40 | TFS Installation completed successfully!!! more configurations will be made later. |
| 13:40 | Our DBA crated backup task – and ran it |
| 13:52 | Backup completed |
| 14:46 | Finished to switch folders to branches |
| 16:27 | VS 2010 installed on Build machine |
| 16:30 | Proxy server (Beta) configured to work with the upgraded instance |
| 16:42 | Team Build installed |
To be continued after Shabbat…
| 19:20 | Trying to run build It failed because of workspace problem Removing workspaces and try again |
| 19:40 | Build failed because of ResGen bug of long command lines. known issue – will be solved by removing all not needed resx files. |
| 20:50 | Build still fails because workspace still connected to old URL. Need to install compatible Team Explorer for VS 2008 |
| 21:11 | Build passed! |
| 22:30 | Important build passed (after switching TF path in environment variables) |
| 22:56 | Configured our internal extension web service for TFS |
| 23:08 | Working Items customized event works |
| 00:40 | Timesheet implementation (in TFS WorkItems) works as well |
| 01:00 | Diff build doesn’t work correctly |
| 01:11 | new WIs and Categories Imported |
| 02:30 | Diff build still do make problems: the order of the build is incorrect, and the incremental flag is ignored. Need to see if the diff will overcome this or not |
| 03:45 | Diff still a problem – will be left for another day |
| 04:07 | Sending mail to all users |
Monday, January 11, 2010
Trying to leave Skype conversation…
[23:20:10] X: Do you know you can leave a conversation? Go to -> Conversation (in the menu)--> Leave Conversation...
[23:20:17] *** X has left ***
[23:20:54] *** Y added X***
[23:20:56] Y: do you know we can add you back?
[23:21:16] Z: lol
[23:22:25] X: Do you know I can close my Skype?
[23:22:50] Y: lol
p.s. you can also write /leave in the conversation - which will cause you to leave too.
Saturday, December 05, 2009
HTTP Error 504, sc-win32-status 64, Alteon and the new employee’s phone
It started in the morning – in our QA environment.
Our topology (simplified – there are lots more connected to the LAN switch in the middle):
The QA engineer is trying to test web application (installed both on IIS server 1 & IIS Server2) via our a load balancer (Alteon in our case).
Yesterday, everything worked fine.
Today, instead of getting the homepage of the web application, we got HTTP error 504.
Quick look at the List of HTTP status codes in Wikipedia:
504 Gateway Timeout - The server was acting as a gateway or proxy and did not receive a timely request from the downstream server.
Now in English:
The Alteon, which works as a gateway between the user and the 2 web applications says it got no response from the IIS server (either 1 or 2 – or both).
a quick check of the web application – by approaching their IPs directly – showed no problem at all.
They were responsive as usual.
We have asked whether there were any network changes lately, but nobody couldn’t remember any change at all.
A deeper look at the IIS logs showed a request from the Alteon did accepted.
The HTTP response was 200 – which was strange – as the Alteon kept telling it got no response.
Then we have noticed, that also the HTTP status is 200, the sc-win32-status was 64.
We already familiar with sc-win32-status 64. but if you don’t:
The specified network name is no longer available
Which means that IIS got the request submit the response successfully to the network – only to find out it can’t reach the client who sent the request.
In our case, this error says the Alteon can see and send requests to the IIS servers, but the responses can’t reach the Alteon back.
So we changed the IP address of the Alteon Load Balancer – and – Hooray!!! it worked.
We re asked about any change in the network, this time we have specified the IP number…
And this was the real topology:
This morning a new employee join the company, and we have IP phones.
The IP Address which was given to the new employee’s IP phone was x.x.x.56…
Monday, November 16, 2009
WCF middle tier client – ClientBase proxy (svcutil) vs. ChannelFactory
After running with WCF based SOA in our production for a year or so, we have been working on our next major version. We have added some new services lately.
in the load test, we encounter the following error:
System.Net.Sockets.SocketException: Only one usage of each socket address (protocol/network address/port) is normally permitted
After some investigation (See Durgaprasad Gorti's WebLog) we understood that as we load huge amount of request on our middle tier application (actually ASP.NET web service) it send lots of requests to other services in our distributed system. We have used TCPView to see what’s going on with the ports – and saw that each request have opened a new client port – which was never reused until the OS released it 4 minutes later.
We have changed the TCPTimedWaitDelay value in the registry to 30 seconds – and it solve the problem.
we decided, however, that this is not a real solution – just a workaround. the real problem is the no reuse of ports – and we need to solve it –as we don’t know when we will hit the wall next time.
Our code used Client Proxy generated by Svcutil.exe, and we have created a new client each time a request was needed:
System.ServiceModel.ClientBase<T> clientProxy = Activator.CreateInstance(proxyType, new object[] { endpoint.Binding, endpoint.Address }) as System.ServiceModel.ClientBase<T>;
Now, one of our developers said that in .NET 3.5 (actually .NET 3.0 SP1) Microsoft has added some kind of caching which should have solved this issue.
As we were running on .NET 3.5 SP1 environment – this clearly hasn’t solved the problem. So we went to look for the reason. It hasn’t took long to find Wenlong Dong's Blog about “Performance Improvement for WCF Client Proxy Creation in .NET 3.5 and Best Practices”.
It was very clear from his blog, that we fail to use the built in caching because we use a constructor with Binding as a parameter. Actually, even if weren’t used a constructor that disable the caching – it was disabled because we accessed the ChannelFactory property too:
clientProxy.ChannelFactory.Endpoint.Behaviors.Add(epb);
We wanted to fix this by calling a different constructor and by adding the behavior to our configuration, but we couldn’t, as our configuration isn’t based on config file, but is read from a central DB – and there is no constructor that doesn’t read configuration from config files (there should be such constructor in 4.0 – See this new constructor).
You can work around this in various ways, like this one by Pablo M. Cibraro.
Read also in great details how the client actually work – at What WCF client does
Anyway, we have decide to different direction.
If you read Microsoft’s Middle-Tier Client Applications, you’ll see 2 options:
- Cache the WCF client object and reuse it for subsequent calls where possible.
- Create a ChannelFactory object and then use that object to create new WCF client channel objects for each call.
Clearly, the first option… is not an option, as it talks about subsequent calls.
We need robust multi threaded middle tier.
So we were left with only one valid solution – to use ChannelFactory instead of the Client proxy, and to cache it (actually redoing the caching mechanism Microsoft added for the ClientBase).
We can succeed caching in the place the framework fails, because we allow ourselves to take some assumptions regarding the reuse of the same ChannelFactory.
We use the cached factory (if not exist – we create one) to return a Typed channel by calling its CreateChannel(). This open a new socket for each request, unless already used socket were closed and available for reuse (You should of course close each created channel, by calling its Close() method).
This is what we wanted.
The load test finished successfully.
Thanks Daniel, Michael & Eyal.
Thursday, October 29, 2009
Convert folder to branch failure - Deleted folder
The error is TF203028: You cannot create a branch at $serverPath because a branch already exists at $SubFolderServerPath.
You can fix this by converting this sub folder back to folder (as you can't have a branch inside a branch)
You might however don't see any such branch folder.
In this case - just set Visual Studio to show deleted items in Team Explorer.
Then you will be able to convert this sub folder into a folder - which will fix the problem
Bypass Gated Check in from TF Checkin Command Line (TFS 2010)
In Team System 2010, Microsoft added long wanted feature – Gated Check in.
This feature lets you promise you won’t break the build by your check in, by converting the check in into a shelveset – running a build with your changes, and only if the build succeed – it check in the code on behalf of you.
It gives you of course the option to bypass this behavior (if you have the required permission).
My problem started in the fact that during the build process itself – we perform a check in…
So this check in fail with the message (and exited with code 1):
Your check-in has been placed into shelveset Gated_XXXXXXX;Domain\User and submitted for validation by build definition \Project-Name\Build-Definition.
I have looked at MSDN for TF Checkin flag that will replace the GUI option to bypass the gated build:
But as you can see, no new option was added.
Luckily, I have run also the TF checkin /? command:
And as you can see, there are few new options here:
- shelveset
- bypass
- login
Those 3 new options came to support this “Gated Check in” feature.
I was actually looking for the /bypass option – which, what a surprise, bypass this feature.
I hope that this MSDN document will be updated soon.
Monday, October 26, 2009
Upgrade from TFS 2008 to TFS 2010 Beta 2 – long wait (literally)
Yes! We have finally got the TFS 2010 Beta 2!!!
I have restored our production Team Foundation Server Databases into a clean computer.
The setup was very easy – just selecting Upgrade flow, and filling some details.
Then it arrived to the most critical part – upgrading the DBs themselves.
It took almost 6 hours…
but… it finished successfully!
So the proof of concept for upgrading has proofed the concept!
Now of course, we still need to see how to upgrade our customized CMMI process – to have all the new staff Microsoft put in the new one. I guess we will use Allen Clark’s Enabling New Features of Visual Studio Team System 2010 Beta 1 in Upgraded Team Projects
