Real Incremental Build - Part 4 – Compare Assemblies using ILDASM

Part 1 – for motivation.
Part 2 – Plan for getting only new/updated files.
Part 3 – Out of the box Incremental build with Team Build.

In last part we succeed getting only new/updated files except binaries which were regenerated also no actual change was made to their code – because some reference in their dependency tree was changed.

We want only real updated assemblies.

Again, the trivial solution doesn’t work:

If you try to compare each assembly with its equivalent in the original build you’ll find that each time you build a project, the assembly is different! (even if no code was changed)

Here comes ILDASM (MSIL Disassembler) to the rescue.

This tool takes and assembly (either DLL or EXE) and can create a text file containing all structure and actual IL code of the file.

If you compare 2 disassembled files which should be identical you’ll probably see the following different lines:

1. Time-date stamp
   This parameter is regenerated each time – so it will be different every time.
   
2. MVID
   This is a generated GUID – Regenerated each time.
   
3. .ver
   Lines with .ver arr declarations of the specific version of referenced assemblies.
   Those lines would be changed if some referenced assemblies has changed their versioning.
   This happens due to the use of * in the AssemblyVersion attribute in AssemblyInfo.cs
   You need to decide whether this is a breaking change from your point of view – or not.
   In my case – we don’t work with Signed DLLs, and I don’t mind what is the specific version. 
   
4. PrivateImplementationDetails
   
   I don’t know for sure what that means, but from my experience it changed without any real change happened.
   
5. WARNING:
   
   Those lines happen to be changed.

So here the actual process:

1. You iterate through all the remaining files.
   
2. If the file’s extension is either .dll or .exe – you run ILDASM on this file, . Those are the parameters I used, but to tell the truth – I haven’t investigate them much:
   
   ildasm.exe  assemblyFileName /OUT=out.txt  /ALL /RAWEH  /SOURCE /LINENUM /CAVERBAL /NOBAR /UTF8 /TYPELIST
   
   Run it second time for the original assembly file too.
   
3. Compare the 2 output files (I just iterate through the lines in both files) and ignore the changes which have been described above. (Time-date stamp, MVID, etc.)
   
   I would jump to the next line which doesn’t contain “WARNING:” in case I meet such a line.
   This is because I have seen cases were only one of the assemblies contained such a line, and then it will break you comparison if you not jump to the next valid line.
   
4. If the files are equal – delete the assembly from the drop location.

The result of this procedure is: Diff package containing only new/updated files

Easy it was, wasn’t it?

Real Incremental Build - Part 3 – Out of the box with Team Build

See Part 1 – for motivation.
See Part 2 – Plan for getting only new/updated files.

Ok, so how do we actually perform our plan?

We will write an application which does the following

1. Run Build with the original source:
   
   See Building a Specific Version with Team Build 2008.
   to put here the bottom line – add the following parameter to the command-line arguments:
   /p:GetVersion=version (where version can be C### – which means – Changeset number ###)
   
   for doing this programmatically, use IBuildServer.GetBuildDefinition(teamProject, name) and call CreateBuildRequest.
   
   The following line sets the command line to get the version you want:
   BuildRequest.CommandLineArguments = "/p:GetVersion=" + OriginalVersionSpec;
   
   Call QueueBuild(BuildRequest) to actually start the build. 
   
2. Wait till the build ends:
   
   You can create a Timer and check the build status each time - QueuedBuild.Status
   You’ll need to refresh the build object – using - QueuedBuild.Refresh() before.
   Save the finish time for later use.
   
3. Get changes only:
   
   This is the actual incremental build (all previous steps were preparations).
   You do this the same way like the build before (with the /p:GetVersion).
   But this time, you add another parameter: IncrementalBuild
   
   The full parameter string will look like this: (see the semicolon separator) 
   
   "/p:IncrementalBuild=true;GetVersion=" + CurrentVersionSpec
   
4. Rebuild:
   
   Queue this build as well, and again wait for it to finish.
   
5. Delete old files:
   
   Iterate through all files recursively in the drop location of the incremental build.
   Delete all files with LastWriteTime older than the finish time of the original build.

So now it seems we have what we wanted.

In the Drop location of the incremental build – only newer files will remain.
All files which weren't changed (or executables which weren’t regenerated due to no code change) – were deleted – leaving new/updated files only.

Or is it?

If you’ll look at the results – you’ll see lots of binaries (DLLs and EXEs) which shouldn’t be there.
No code change was done in their projects.
So why are they here?

Because at least one of their references assemblies has changed…

So we succeed partially:
All file types except assemblies – we have updated files only.
Assemblies – we have all files which had a change somewhere in their dependency tree.

Back to square one?

Solution in part 4...

Real Incremental Build - Part 2 – Getting only new/updated files

see part 1 for motivation.

So how do you prepare a package with only the new and updated files?
(whether those are binaries, images, aspx files, resource files, etc.)

The simplistic answer would be to ask the developer which files he touched, and either take them themselves or their products (e.g. the assembly which was generated from the source file).

However there could be lots of developers involved who changed multiple files each.
Try to keep a trace of all those files…

You can use your source control to find all new and changes files, of course, but that still requires someone to sit and analyze which of them can be taken as is, and which of them generates changes in end products.

The knowledge of what file update ends up in a new version of which dll or exe sits inside the project files. So you must use this knowledge automatically.

Which means… Build. 

But wouldn’t a build gives us the whole package again?
No. at least not if you use Incremental Build.

Remember when you change some code in visual studio after you already complied big solution?
the next build finishes faster.
Why? because it rebuild only those projects that have been changed.

So let’s see how can we use it:

1. Build the original source (which already in use in production).
2. Write down the build finish time.
3. Get only the changes from source control (all changes from last original check in till last fix).
4. Build again.
5. Get only new files (newer then the original’s finish time) – or – delete all older files.

How to do it technically?

See part 3.

Real Incremental Build - Part 1 – Motivation for small deployment packages

You have deployed version x of your compiled application to your production environment.
Some bugs were resolved and you want to deploy a fix.

You have 2 options:

1. Deploy a full updated build of the application.
2. Deploy differential package only.

Now, Option 1 will clearly work.
You run your build, take everything from the drop location, and deploy it.

Pros:

• You're sure that you won't missed anything.
• All files are compatible with each other.

Cons:

• The package will have big size (lots of MB – depends on your application size)

At first, the size of the patch looks unimportant.

But if you take into account the time it will take to upload it to your production environment and distributed it between your hundreds of applications instances – this would be unwise not to consider this factor. Or if you let users to download this patch from your website – I guess you’ll want to minimize its size to save your bandwidth usage.

on the next parts we will go from this globally described scenario into more detailed one – based on Microsoft TFS (Team foundation server) source control & Team Build. We will try to show how to create a delta package contains only the new and updated binaries and files. We will meet some implantation problems – and hopefully overcome them.

See part 2.

Tamar Yifrach – תמר יפרח - First Music Album

My sis, Tamar Yifrach, has created a wonderful Folk/Pop/Jewish/Hebrew music album:

The album name is: “תמיד היה מזה קצת בעולם” or “There was always a bit from that in the world”.

You can listen to some tracks from the disc in Tamar Yifrach on MySpace
And for the Hebrew readers – her name is תמר יפרח.

Hope you’ll enjoy it.

I did.

Microsoft Songsmith is fun (with some suggestions)

I have download Microsoft Songsmith today.

I guess most people begins their playing with this application by singing known songs, excepting it to perfectly redo the original music of their selected song.

So either I'm a totally bad performer, or it just doesn’t work.

the fun begins when you let yourself inventing songs with it.
The even greater fun is to do it together with someone!

My suggested improvements:

• Let the algorithm decide whether to put one or two chords in each bar separately (instead of setting it per the whole song)
• Let me change the “Happy” or “Jazzy” per section of the song.

Hope you’ll enjoy it.

.NET Framework 3.5 Family Update - My Peek inside the .NET 2 update

UPDATE:

KB 959209 has appeared! (was published on December 20, 2008).
This KB point you to 3 other KBs for the 3 parts of the Family Update:

• 958481 – .Net 2.0 SP2 update
• 958483 - .Net 3.0 SP2 update
• 958484 - .Net 3.5 SP1 update

(Those docs are very similar each to the other, telling about the same issues)
Better late than never.

ORIGINAL POST: 

As I stated in my previous entry, Microsoft has uploaded this Family Update, with the usual missing of a description document what went into this update.

The only thing they said was:

The Microsoft .NET Framework 3.5 Family Update provides compatibility roll-up updates for customer reported issues found after the release of Microsoft .NET Framework 3.5 SP1

So I have decided to peek inside.

Notice:
I don't take any responsibility on the following information. I might have done something totally wrong in the way I have extracted the update and got wrong info from within. You take all the risk on yourself if you use this info.

Steps:

Now that we done with those legal issues, what I have done is extract the .msp file from within the the NDP20SP2-KB958481-x86.exe.

Then, using MSIX tool, I have extract the files from the .msp file.

Then, extract the patch.cab file.

Next step was renaming the dlls I have found to have the correct extension (which was missing)

Next, for each dll (only managed), using Reflector, disassemble and export the source files.
Doing the same last action to the original dlls from the .NET GAC.

Using Araxis Merge to compare the source.

Please remember that I have done the investigation only for the .NET 2.0 SP2 Family update.

Non managed dlls in the update: (can't say what was changed there)

• aspnet_wp.exe
• mscordacwks.dll
• mscorjit_dll
• webengine.dll

Managed dlls:

• mscorlib.dll
• System.Data.OracleClient
• System.Xml
• System.Web

Main changes in those Managed dlls:

(I will just write the changes in short, I don't have more info to share- this would required days of reading Microsoft's code).

System.Web:

• Better handling of IIS Rewrite Module in multiple classes(coockieless security, IIS7 worker request, HTML Form, Client Path)
• Process Custom Browser Files - ignores hidden files in the directory (at least this what it's look like)
• Something in state cache handling.
• Change in Sort entries in control builder.

System.Xml

• xml CanWrite check instead of GetSetMethod in one method.

System.Data.OracleClient

• RollbackDeadTransaction and some more Transactions code was added to a method .

mscorlib

• Handling Delegate & MulticastDelegate Combine null issue better

the Version was changed from 2.0.50727.3053 to 2.0.50727.3082.

wow, this was long.
Hope you enjoy it.

I still wait for the missing KB doc from Microsoft...

If it won't come soon, you might find here the same analysis to the 2 other updates (I don't promise).