.Net ramblings
# Thursday, 29 December 2005
Sending files in chunks with MTOM Web Services and .NET 2.0
just posting my code-project article on http chunking web services with MTOM here for reference.  by the way it has been updated on codeproject...

Screenshot of windows forms client, uploading a file

Introduction

In trying to keep up to speed with .NET 2.0, I decided to do a .NET 2.0 version of my CodeProject article "DIME Buffered Upload" which used the DIME standard to transfer binary data over web services. The DIME approach was reasonably efficient but the code is quite complex and I was keen to explore what .NET 2.0 has to offer. In this article, I use version 3.0 of the WSE (Web Service Enhancements) which is available for .NET 2.0 as an add-in, to provide a simpler and faster method of sending binary data in small chunks over HTTP web services.

Background

Just a recap on why you may need to send data in small chunks at all: if you have a large file and you want to send it across a web service, you must understand the way it all fits together between IIS, .NET, and the web service call. You send your file as an array of bytes, as a parameter to a web service call, which is all sent to the IIS web server as a single request. This is bad if the size of the file is beyond the configured MaxRequestLength of your application, or if the request causes an IIS timeout. It is also bad from the point of view of providing feedback of the file transfer to the user interface because you have no indication how the transfer is going, until it is either completed or failed. The solution outlined here is to send chunks of the file one by one, and append them to the file on the server.

There is an MD5 file hash done on the client and the server to verify that the file received is identical to the file sent.

Also, there is an upload and download code included in this article.

Adventures with MTOM

MTOM stands for SOAP "Message Transmission Optimization Mechanism" and it is a W3C standard. To use it (and to run this application), you must download and install WSE 3.0, which includes MTOM support for the first time. If you look in the app.config and web.config files in the source code, you will see sections referring to the WSE 3 assembly, and a messaging clientMode or serverMode setting. These are necessary to run MTOM in the application.

The problem with DIME was that the binary content of the message was sent outside the SoapEnvelope of the XML message. This meant that although your message was secure, the Dime Attachment may not be secure. MTOM fully complies with the other WS-* specifications (like WS-Security) so the entire message is secure.

It took me a while to realise that when MTOM is turned on for the client and the server, WSE automatically handles the binary encoding of the data in the web service message. With DIME and WSE 2.0, you had to configure your app for DIME and then use DimeAttachments in your code. This is no longer necessary, you just send your byte[] as a parameter or return value, and WSE makes sure that it is sent as binary, and not padded by XML serialization as it would be in the absence of DIME or MTOM.

How it works

The web service has two main methods, AppendChunk is for uploading a file to the server, DownloadChunk is for downloading from the server. These methods receive parameters for the file name, the offset of the chunk, and the size of the buffer being sent/received.

The Windows Forms client application can upload a file by sending all the chunks one after the other using AppendChunk, until the file has been completely sent. It can do an MD5 hash on the local file, and compare it with the hash on the file on the server, to make sure the contents of the files are identical. The download code is very similar, the main difference is that the client must know from the server how big the file is, so that it can know when to stop requesting chunks.

A simplified version of the upload code is shown below (from the WinForms client). Have a look in the code for Form1.cs to see the inline comments + the explanation of the code. Essentially, a file stream is opened on the client for the duration of the transfer. Then the first chunk is read into the Buffer byte array. The while loop keeps running until the FileStream.Read() method returns 0, i.e. the end of the file has been reached. For each iteration, the buffer is sent directly to the web service as a byte[]. The 'SentBytes' variable is used to report progress to the form.

using(FileStream fs = new FileStream(LocalFilePath, FileMode.Open, FileAccess.Read))
{
int BytesRead = fs.Read(Buffer, 0, ChunkSize);
while(BytesRead > 0 && !worker.CancellationPending)
{
ws.AppendChunk(FileName, Buffer, SentBytes, BytesRead);
SentBytes += BytesRead;
BytesRead = fs.Read(Buffer, 0, ChunkSize);
}
}

Example of the BackgroundWorker class in .NET 2.0

.NET 2.0 has a great new class called 'BackgroundWorker' to simplify running tasks asynchronously. Although this application sends the file in small chunks, even these small chunks would delay the WinForms application and make it look crashed during the transfer. So the web service calls still need to be done asynchronously. The BackgroundWorker class works using an event model, where you have code sections to run for DoWork (when you start), ProgressChanged (to update your progress bar / status bar), and Completed (or failed). You can pass parameters to the DoWork method, which you could not do with the Thread class in .NET 1.1 (I know you could with delegates, but delegates aren't great for thread control). You can also access the return value of DoWork in the Completed event handler. So for once, MS has thought of everything and made a very clean threading model. Exceptions are handled internally and you can access them in the Completed method via the RunWorkerCompletedEventArgs.Error property.

The code shown below is an example of the ProgressChanged event handler:

private void workerUpload_ProgressChanged(object sender, ProgressChangedEventArgs e)
{
// update the progress bar and status bar text
this.toolStripProgressBar1.Value = e.ProgressPercentage;
this.statusText.Text = e.UserState.ToString();
// summary text is sent in the UserState parameter
}

I have used four BackgroundWorker objects in the application:

  • one to manage the upload process,
  • one to manage the download process,
  • another to calculate the local MD5 file hash in parallel while waiting for the server result,
  • and another to download the list of files in the Upload server folder to allow the user to select a file to download.

The reason I use a BackgroundWorker object for each task is because the code for each task is tied in to the events for that object.

A good example of Thread.Join()

When the upload or download is complete, the client asks for an MD5 hash of the file on the server, so it can compare it with the local file to make sure they are identical. I originally did these in sequence. But it can take a few seconds to calculate the result for a large file (anything over a few hundred MB), so the application was waiting five seconds for the server to calculate the hash, and then five more seconds for the client to calculate its own hash. This made no sense, so I decided to implement a multi-threaded approach to allow them to run in parallel. While the client is waiting on the server, it should be calculating its own file hash. This is done with the Thread class, and the use of the Join() method which blocks execution until the thread is complete.

The code below shows how this is accomplished:

// start calculating the local hash (stored in class variable)
this.hashThread = new Thread(new ThreadStart(this.CheckFileHash));
this.hashThread.Start();

// request the server hash
string ServerFileHash = ws.CheckFileHash(FileName);

// wait for the local hash to complete
this.hashThread.Join();

if(this.LocalFileHash == ServerFileHash)
e.Result = "Hashes match exactly";
else
e.Result = "Hashes do not match";

There is a good chance that the two operations will finish at approximately the same time, so very little waiting around will actually happen.

Performance compared with DIME

I found that MTOM was about 10% faster than DIME in my limited testing. This is probably to do with the need to package up each chunk into a DIME attachment, which is no longer necessary with MTOM. I was able to upload files of several gigabytes in size without problems.

Obviously, there is an overhead with all this business of reading file chunks and appending them, so the larger the chunk size, the more efficient your application will be. It should be customised based on the network and the expected size of files. For very small files, it is no harm to use small chunk sizes (e.g., 32 Kb) because this will give accurate and regular feedback to the user interface. For very large files on a fast network, consider using 4000 Kb to make good use of the bandwidth and reduce the File Input/Output overhead. If you want to send chunks larger than 4 MB, you must increase the .NET 2.0 Max Request Size limit in your web.config.

Conclusions

Feel free to use this code and modify it as you please. Please post a comment for any bugs, suggestions, or improvements. Enjoy!


Thursday, 29 December 2005 20:34:15 (GMT Standard Time, UTC+00:00)  #    Comments [24]  .Net General | .Net Windows Forms | Asp.Net

# Thursday, 22 December 2005
MTOM error: The request failed with HTTP status 415: Unsupported media type
I'm new to MTOM, and i encountered this error while developing a WSE 3.0 application with .net.
The request failed with HTTP status 415: Unsupported media type

Explanation 1 - Basic Configuration Error

After a lot of digging around and comparing my code with the MTOM sample bundled with WSE3, i found out it was because i was missing a setting called "soapServerProtocolFactory" in my web.config, shown below.  I didn't think i needed to have a Protocol Factory (whatever that is!), but apparently I do. It works fine now that i added that section.

This is what my web.config looks like:
<?xml version="1.0" encoding="utf-8"?>
<configuration>
<configSections>
<section name="microsoft.web.services3" type="Microsoft.Web.Services3.Configuration.WebServicesConfiguration, Microsoft.Web.Services3, Version=3.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" />
</configSections>
<appSettings>
<add key="UploadPath" value="Upload" />
</appSettings>
<system.web>
<webServices>
<soapServerProtocolFactory type="Microsoft.Web.Services3.WseProtocolFactory, Microsoft.Web.Services3, Version=3.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" />
<soapExtensionImporterTypes>
<add type="Microsoft.Web.Services3.Description.WseExtensionImporter, Microsoft.Web.Services3, Version=3.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" />
</soapExtensionImporterTypes>
</webServices>
<compilation>
<assemblies>
<add assembly="Microsoft.Web.Services3, Version=3.0.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35" />
</assemblies>
</compilation>
</system.web>
<microsoft.web.services3>
<messaging>
<mtom serverMode="optional" />
</messaging>
</microsoft.web.services3>
</configuration>
My app.config for the Windows Client looks like this:
<?xml version="1.0" encoding="utf-8"?>
<configuration>
<configSections>
<section name="microsoft.web.services3" type="Microsoft.Web.Services3.Configuration.WebServicesConfiguration, Microsoft.Web.Services3, Version=3.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" />
<sectionGroup name="applicationSettings" type="System.Configuration.ApplicationSettingsGroup, System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089">
<section name="UploadWinClient.Properties.Settings" type="System.Configuration.ClientSettingsSection, System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" requirePermission="false" />
</sectionGroup>
</configSections>
<microsoft.web.services3>
<messaging>
<mtom clientMode="On" />
</messaging>
</microsoft.web.services3>
<applicationSettings>
<UploadWinClient.Properties.Settings>
<setting name="UploadWinClient_MTOM_WebService_MTOM" serializeAs="String">
<value>http://localhost:1423/UploadWebClient/MTOM.asmx</value>
</setting>
</UploadWinClient.Properties.Settings>
</applicationSettings>
</configuration>

Explanation 2 - Strange Behaviour with Web Service instances...

if you're configuration is already correct, then the following may be of interest.  i came across this error in a winforms application, that has a class library with an MTOM web service in it.  my code had something like this:
myClassLibrary.WebService = this.WebService;
i was doing this because i had an instance of the web service in the win form, authenticated with a cookie in its CookieContainer property, and i wanted to replace the web service that the class library was using with the one from my win-form.  however this caused the 415 error to appear sporadically all over the place.  the fix was to just replace the CookieContainer instead of the entire web service instance.  like so:
myClassLibrary.WebService.CookieContainer = this.WebService.CookieContainer;

now i get no errors.  weird huh?

Thursday, 22 December 2005 16:13:34 (GMT Standard Time, UTC+00:00)  #    Comments [7]  Asp.Net

# Wednesday, 21 December 2005
GridView Databinding error "Field or property ... not found on the selected data source"

A field or property with the name 'xyz' was not found on the selected data source

I get this error when i bind a GridView to an array of custom objects with public variables. 
The recommended pattern for designing classes is to have private variables with public properties with get/set etc, but for my current application i'm not bothering with that. Apparently because of the Reflection methods used in GridView databinding, you have to have properties to bind fields in the GridView to your object fields. 

bit of a pain.


Wednesday, 21 December 2005 15:51:28 (GMT Standard Time, UTC+00:00)  #    Comments [3]  Asp.Net

# Tuesday, 20 December 2005
GridView - accessing row information inside RowCommand
in .net 1.1, the GridCommandEventArgs (or whatever) used to give access to the row that triggered the command. 
in .net 2.0, it isn't as obvious as that.  Fritz posted an excellent discussion on the matter, and i'm just posting his 2-line solution here for reference.

If your button is a ButtonField, then you can access the index of the item via e.CommandArgument.  This may be enough information to do whatever you need.

If your button is a LinkButton in a TemplateField, you have to add the CommandArgument as an attribute to the LinkButton:
CommandArgument='<%# Eval("id") %>'
Then in the code behind for RowCommand, you can use the following syntax:
string id = (string)e.CommandArgument;
thanks Fritz!

Tuesday, 20 December 2005 17:52:22 (GMT Standard Time, UTC+00:00)  #    Comments [0]  Asp.Net

# Friday, 16 December 2005
GridView with ObjectDataSource, Delete parameters not working...

i'm just experimenting with the gridview control, and i got stuck trying to delete an object with the auto-included delete button.  the delete method would get invoked properly, but the ID parameter was always zero, which means the int parameter was uninitialised.

my problem was that i didn't have the DataKeyNames property set in the Gridview. I just set it to 'ID' and it worked fine then.


Friday, 16 December 2005 14:21:53 (GMT Standard Time, UTC+00:00)  #    Comments [18]  Asp.Net

# Thursday, 15 December 2005
Usability problems with 'directory listing' in .Net 2.0

Microsoft love small fixed fonts for some reason

Correction: this only applies to the Asp.Net development server (v8), so it's not all that important...

i noticed if you browse a folder on a .net 2 virtual directory, you get the directory listing.  but the font styles are set to 8 point verdana which is uncomfortably small for some users on some resolutions.  the fact that they hard-coded it in the css styles means it ignores the browser font-size setting, so it breaks a very important accessibility guideline.  

this is just another reason to use firefox which lets you increase font sizes for fixed sizes. surely MS would be trying to hang on to their user population who still uses IE.  
i looked in the source of the directory listing, and i see they have the file and folder info formatted inside <pre> tags, keeping the tabs aligned nicely.  but i took a copy of the source and removed the font sizes, and the alignment is still fine when you change the browser font size!

come on MS... why the love affair with "px" and "pt" for font sizes?  
i remember gotdotnet.com had fixed sized fonts also.  it's a silly decision in my book, let your users control the font size.  

i've just been experimenting with VS2005 and i like the improved support for xhtml and accessibility, but lots of people actually use directory listings, so why not make them accessible also?


Thursday, 15 December 2005 17:18:34 (GMT Standard Time, UTC+00:00)  #    Comments [0]  Asp.Net

Serializing an object in Xml, with .Net 2.0
i was experimenting with how .Net 2.0 does xml serialization of objects, and i got it serializing nicely with the following code. 
XmlSerializer xs = new XmlSerializer(typeof(PageCollection));
xs.Serialize(fs, pages); // 'fs' is a FileStream to my xml file, and 'pages' is a collection class of objects

the problem was when i tried to deserialize it, like so:
XmlSerializer xs = new XmlSerializer(typeof(PageCollection));
pages = xs.Deserialize(fs) as PageCollection;

i got this error:

xmlns=''> was not expected

i found this post on google which described the same errors, and a workaround (by adding an empty namespace) but it didn't work for me.  perhaps it is a difference in the serialization process with with .Net 2.0.  what fixed it for me was setting an XmlRootAttribute for the class i was serializing. like so:
[XmlRootAttribute("CmsPages", Namespace = "http://www.whatever.com/Cms", IsNullable = false)]
hope this helps someone else out there with the same problem.
Thursday, 15 December 2005 13:28:01 (GMT Standard Time, UTC+00:00)  #    Comments [0]  .Net General | Asp.Net | Database

# Sunday, 11 December 2005
How Not to Develop Software

Usability quote of the day

i'm doing some contract work developing a content management system for a software/design company, and the boss doesn't think it's a good idea to involve the client at all in the process, basically they want a localizable CMS with built-in publishing / approval process, and they are going to be given something along those lines (which i'm hired to develop) and be told how to use it.  That's about as far as the communication goes with the client.  this is bad enough, but yesterday there was a certain design decision we had to make that would seriously affect the client and how they can use the system, and i suggested asking the client for their opinion...

"  oh jeez no!  "

this was the response from the boss of this un-named company, "don't talk to them until the thing is finished ".  i had to hold back the laughter at such a dim view of developing software.  i have a distinct feeling the client would like it done differently but what can you do!  i should buy this guy a ticket to a jakob nielsen conference on usability...


Sunday, 11 December 2005 16:28:44 (GMT Standard Time, UTC+00:00)  #    Comments [1]  General

# Friday, 09 December 2005
Micrsoft Word: System Call Failure
I laughed out loud when this message popped up, typical of Microsoft - the inventor of useful error messages. 
i was in outlook, using word as the email editor, and it wouldn't save draft or send the message at all.  i had to kill the winword and outlook processes and restart outlook. luckily it let me copy/paste into notepad so i didn't lose anything.

no idea what caused it.


Friday, 09 December 2005 19:31:13 (GMT Standard Time, UTC+00:00)  #    Comments [0]  General

# Friday, 02 December 2005
ADO.NET timeouts when filling a DataAdapter in the middle of a transaction

I had a transaction comprised of about 5 commands, and in the middle of it, i needed to do a SELECT on a table to know what values to insert for one of the commands.  I encountered a timeout just after i called Fill on the DataAdapter.  it took me a while to figure out that it was because the transaction had already taken out a lock on the table i was selecting from.  makes sense now, but i spent ages on it!  just posting it here in case anyone else runs into the same problem.


Friday, 02 December 2005 12:26:22 (GMT Standard Time, UTC+00:00)  #    Comments [0]  .Net General | .Net Windows Forms | Asp.Net | Database