.Net ramblings
# Thursday, 29 December 2005
Sending files in chunks with MTOM Web Services and .NET 2.0
just posting my code-project article on http chunking web services with MTOM here for reference.  by the way it has been updated on codeproject...

Screenshot of windows forms client, uploading a file

Introduction

In trying to keep up to speed with .NET 2.0, I decided to do a .NET 2.0 version of my CodeProject article "DIME Buffered Upload" which used the DIME standard to transfer binary data over web services. The DIME approach was reasonably efficient but the code is quite complex and I was keen to explore what .NET 2.0 has to offer. In this article, I use version 3.0 of the WSE (Web Service Enhancements) which is available for .NET 2.0 as an add-in, to provide a simpler and faster method of sending binary data in small chunks over HTTP web services.

Background

Just a recap on why you may need to send data in small chunks at all: if you have a large file and you want to send it across a web service, you must understand the way it all fits together between IIS, .NET, and the web service call. You send your file as an array of bytes, as a parameter to a web service call, which is all sent to the IIS web server as a single request. This is bad if the size of the file is beyond the configured MaxRequestLength of your application, or if the request causes an IIS timeout. It is also bad from the point of view of providing feedback of the file transfer to the user interface because you have no indication how the transfer is going, until it is either completed or failed. The solution outlined here is to send chunks of the file one by one, and append them to the file on the server.

There is an MD5 file hash done on the client and the server to verify that the file received is identical to the file sent.

Also, there is an upload and download code included in this article.

Adventures with MTOM

MTOM stands for SOAP "Message Transmission Optimization Mechanism" and it is a W3C standard. To use it (and to run this application), you must download and install WSE 3.0, which includes MTOM support for the first time. If you look in the app.config and web.config files in the source code, you will see sections referring to the WSE 3 assembly, and a messaging clientMode or serverMode setting. These are necessary to run MTOM in the application.

The problem with DIME was that the binary content of the message was sent outside the SoapEnvelope of the XML message. This meant that although your message was secure, the Dime Attachment may not be secure. MTOM fully complies with the other WS-* specifications (like WS-Security) so the entire message is secure.

It took me a while to realise that when MTOM is turned on for the client and the server, WSE automatically handles the binary encoding of the data in the web service message. With DIME and WSE 2.0, you had to configure your app for DIME and then use DimeAttachments in your code. This is no longer necessary, you just send your byte[] as a parameter or return value, and WSE makes sure that it is sent as binary, and not padded by XML serialization as it would be in the absence of DIME or MTOM.

How it works

The web service has two main methods, AppendChunk is for uploading a file to the server, DownloadChunk is for downloading from the server. These methods receive parameters for the file name, the offset of the chunk, and the size of the buffer being sent/received.

The Windows Forms client application can upload a file by sending all the chunks one after the other using AppendChunk, until the file has been completely sent. It can do an MD5 hash on the local file, and compare it with the hash on the file on the server, to make sure the contents of the files are identical. The download code is very similar, the main difference is that the client must know from the server how big the file is, so that it can know when to stop requesting chunks.

A simplified version of the upload code is shown below (from the WinForms client). Have a look in the code for Form1.cs to see the inline comments + the explanation of the code. Essentially, a file stream is opened on the client for the duration of the transfer. Then the first chunk is read into the Buffer byte array. The while loop keeps running until the FileStream.Read() method returns 0, i.e. the end of the file has been reached. For each iteration, the buffer is sent directly to the web service as a byte[]. The 'SentBytes' variable is used to report progress to the form.

using(FileStream fs = new FileStream(LocalFilePath, FileMode.Open, FileAccess.Read))
{
int BytesRead = fs.Read(Buffer, 0, ChunkSize);
while(BytesRead > 0 && !worker.CancellationPending)
{
ws.AppendChunk(FileName, Buffer, SentBytes, BytesRead);
SentBytes += BytesRead;
BytesRead = fs.Read(Buffer, 0, ChunkSize);
}
}

Example of the BackgroundWorker class in .NET 2.0

.NET 2.0 has a great new class called 'BackgroundWorker' to simplify running tasks asynchronously. Although this application sends the file in small chunks, even these small chunks would delay the WinForms application and make it look crashed during the transfer. So the web service calls still need to be done asynchronously. The BackgroundWorker class works using an event model, where you have code sections to run for DoWork (when you start), ProgressChanged (to update your progress bar / status bar), and Completed (or failed). You can pass parameters to the DoWork method, which you could not do with the Thread class in .NET 1.1 (I know you could with delegates, but delegates aren't great for thread control). You can also access the return value of DoWork in the Completed event handler. So for once, MS has thought of everything and made a very clean threading model. Exceptions are handled internally and you can access them in the Completed method via the RunWorkerCompletedEventArgs.Error property.

The code shown below is an example of the ProgressChanged event handler:

private void workerUpload_ProgressChanged(object sender, ProgressChangedEventArgs e)
{
// update the progress bar and status bar text
this.toolStripProgressBar1.Value = e.ProgressPercentage;
this.statusText.Text = e.UserState.ToString();
// summary text is sent in the UserState parameter
}

I have used four BackgroundWorker objects in the application:

  • one to manage the upload process,
  • one to manage the download process,
  • another to calculate the local MD5 file hash in parallel while waiting for the server result,
  • and another to download the list of files in the Upload server folder to allow the user to select a file to download.

The reason I use a BackgroundWorker object for each task is because the code for each task is tied in to the events for that object.

A good example of Thread.Join()

When the upload or download is complete, the client asks for an MD5 hash of the file on the server, so it can compare it with the local file to make sure they are identical. I originally did these in sequence. But it can take a few seconds to calculate the result for a large file (anything over a few hundred MB), so the application was waiting five seconds for the server to calculate the hash, and then five more seconds for the client to calculate its own hash. This made no sense, so I decided to implement a multi-threaded approach to allow them to run in parallel. While the client is waiting on the server, it should be calculating its own file hash. This is done with the Thread class, and the use of the Join() method which blocks execution until the thread is complete.

The code below shows how this is accomplished:

// start calculating the local hash (stored in class variable)
this.hashThread = new Thread(new ThreadStart(this.CheckFileHash));
this.hashThread.Start();

// request the server hash
string ServerFileHash = ws.CheckFileHash(FileName);

// wait for the local hash to complete
this.hashThread.Join();

if(this.LocalFileHash == ServerFileHash)
e.Result = "Hashes match exactly";
else
e.Result = "Hashes do not match";

There is a good chance that the two operations will finish at approximately the same time, so very little waiting around will actually happen.

Performance compared with DIME

I found that MTOM was about 10% faster than DIME in my limited testing. This is probably to do with the need to package up each chunk into a DIME attachment, which is no longer necessary with MTOM. I was able to upload files of several gigabytes in size without problems.

Obviously, there is an overhead with all this business of reading file chunks and appending them, so the larger the chunk size, the more efficient your application will be. It should be customised based on the network and the expected size of files. For very small files, it is no harm to use small chunk sizes (e.g., 32 Kb) because this will give accurate and regular feedback to the user interface. For very large files on a fast network, consider using 4000 Kb to make good use of the bandwidth and reduce the File Input/Output overhead. If you want to send chunks larger than 4 MB, you must increase the .NET 2.0 Max Request Size limit in your web.config.

Conclusions

Feel free to use this code and modify it as you please. Please post a comment for any bugs, suggestions, or improvements. Enjoy!


Thursday, 29 December 2005 20:34:15 (GMT Standard Time, UTC+00:00)  #    Comments [24]  .Net General | .Net Windows Forms | Asp.Net

Thursday, 05 January 2006 10:44:50 (GMT Standard Time, UTC+00:00)
Hi,

The article is very useful but onething i am unable to run the web service mtom.asmx can u give me suggestion to run tat.
Ram
Thursday, 05 January 2006 10:46:36 (GMT Standard Time, UTC+00:00)
hi.
you need .Net framework 2.0 and WSE 3.0 installed on the computer. the web site folder needs READ+EXECUTE permissions for the web site identity account (e.g. ASPNET). if you have all these already, then post the error message you are getting.
Tim Mackey
Tuesday, 17 January 2006 07:33:44 (GMT Standard Time, UTC+00:00)
Hi,

Thanks for reply.

Ya fine Tim but I'm unable to upload file from my local system to server.

I mean suppose the web service for file upload is in other server and if I am using tat web service for file upload then the file is uploading using server c:\\xyz.txt but not from my local machine.

Plz help me in this regarding.

Actually i want to upload file without using input type file options.


Thanks & regards
Ram.S
R


Ram
Tuesday, 17 January 2006 08:28:24 (GMT Standard Time, UTC+00:00)
Hi ram,
The code is designed to run normally with the client and server on different computers. I'm not sure if you understand the architecture of this application, it is a winforms client connecting to a web application, for the purposes of transferring files. You mention input type file options, I presume you are referring to the FileInput WebControl with .net 2.0. I'm not using this control anywhere, because this solution will not work for transferring files from asp.net to asp.net. it is only from win-forms to asp.net web service.

I don't know what the problem is then. What error are you getting? Have you got your IIS virtual directory set up correctly? Permissions properly assigned?

Tim
Tim Mackey
Thursday, 26 January 2006 17:40:05 (GMT Standard Time, UTC+00:00)
Good article. I was stumped on MTOM implementation for a while until I saw this. It has been a real help.

Randy
Randy Felts
Sunday, 29 January 2006 02:13:50 (GMT Standard Time, UTC+00:00)
Hi,
Great article.
Talking about the 1.1 framewrk based solution, i´m not able to compile even after install wse 2 sp3.

Whats wrong?
Thanx
Luciano Terra
Friday, 24 March 2006 19:22:30 (GMT Standard Time, UTC+00:00)
Hi Tim

Very good article. Can we use similar approach for ASP.NET client, rather than windows client. How do I browse the local files?. Should I use input type=file or some other technique. Could you share any tips & tricks on this ?

Thanks
Sunny
Sunny
Friday, 24 March 2006 19:25:28 (GMT Standard Time, UTC+00:00)
hi sunny.
no unfortunately it's not that easy. you have already identified the problem, namely that the browser does not have access to the file system except through the file upload control. even with that, it is a single post and it is not possible to do a chunked transfer. java / activeX might be possible.

you can't use this approach from a web client.
good luck
tim
tim
Friday, 31 March 2006 12:13:55 (GMT Daylight Time, UTC+01:00)
Hi,

Hi, ive done a VB.net version of the code. I think ive done it ok as im not a c# programmer. Hope this helps those to VB.net only

Regards

Gary
http://www.garyhowlett.plus.com/MTOMVBandCS.zip
Wednesday, 27 December 2006 05:50:51 (GMT Standard Time, UTC+00:00)
Hello Sir

Does MTOM automatically use built-in WS_Security or do we have to provide some sort of certificate or other security things for it to secure the SOAP message including headers and data etc.?

Best Ragards
Majid
Wednesday, 27 December 2006 08:06:45 (GMT Standard Time, UTC+00:00)
hi majid
try this article "security features in WSE 3"
http://msdn.microsoft.com/webservices/webservices/building/wse/default.aspx?pull=/msdnmag/issues/05/11/securitybriefs/default.aspx

tim
tim
Monday, 02 April 2007 16:32:28 (GMT Daylight Time, UTC+01:00)
Hi Tim,

Your program is very nice and I want to use it under pocket pc. The lib Background worker is not implemented on the pocket pc so it didn't work :( I've tried to include a Open source lib Bacground worker for PPC but it didn't work because this is not the same as the PC. I've tried to change the one use in the PC, but the open source lib doen't containt all the function/properties of the real one. I want to know if you can make working this program under a pocket pc. Do you think it is possible to create a thread an make working your program under the pocket pc ? Plead email me at shibby_crinquer at hotmail.com

Thx very much Tim.
Jean-Philippe C.
Monday, 02 April 2007 16:45:08 (GMT Daylight Time, UTC+01:00)
Hi Jean-Phillipe,
I think it should not be too difficult, as long as it is possible to create a thread using the “Thread” class in PPC. The BackgroundWorker class is merely a more convenient pattern to use for multithreading, when dealing with user interfaces. The alternative is not to do any multithreading and just run the transfer code on the main UI thread. This will obviously block the application during each transfer, but that may be a acceptable.

I hope it works out for you. If you need more advice, I would go to a PPC newsgroup and ask about how to do multithreading on that platform.
Good luck
Tim
tim
Thursday, 12 July 2007 17:45:20 (GMT Daylight Time, UTC+01:00)
I must say that this is an impressive project. All I had to do is point your lib to my hosted web service and had a very stable large file over web service project. I commend you for this example and give credit where it is by all means due.
Thursday, 19 June 2008 08:33:03 (GMT Daylight Time, UTC+01:00)
hi Tim
Its really very nice Article!!TKS
I want to know, whats the maximum limitation for Upload and Download using MTOM?

regards

kamachiSekar
kamachiSekar
Thursday, 19 June 2008 09:02:07 (GMT Daylight Time, UTC+01:00)
Hi kamachiSekar ,
The limitation is really your IIS timeouts. You can configure whatever max request length in web.config, but it isn't practical to bombard a server with 1Gb of data in a single request or something crazy like that. The purpose of this article is to allow large files to be transmitted in small chunks, so I don't really recommend anything above 4Mb for a WAN connection.
Tim
tim mackey
Thursday, 09 July 2009 07:02:03 (GMT Daylight Time, UTC+01:00)
Hi Tim,

I have gone through code its excellent one. I am trying to create sample webapplication and consuming webservice
which given in the code by adding refernce in web application and web.config settings but its not working...

Can you please share code & procedure Webapplication as client instead Windows client

Prasad
Thursday, 09 July 2009 08:34:19 (GMT Daylight Time, UTC+01:00)
hi prasad. this is fundamentally not possible. have a read of the section "Can You Make a Web Client?" on the code project article for an explanation. http://www.codeproject.com/KB/XML/MTOMWebServices.aspx
good luck
tim
tim
Friday, 28 August 2009 07:20:30 (GMT Daylight Time, UTC+01:00)
this article is great, but i want to make same in web application so please please help me out.

Thanks

Sushheide
Friday, 28 August 2009 07:24:54 (GMT Daylight Time, UTC+01:00)
would you help me same for the web application?


this article is great, but i want to make same in web application so please please help me out.

Thanks

Sushheide
sushheide
Friday, 28 August 2009 08:49:17 (GMT Daylight Time, UTC+01:00)
hi susheide. read the previous comments for an explanation of why this is not possible.
don't post duplicate comments.
don't make demands.
tim
Thursday, 03 September 2009 21:33:31 (GMT Daylight Time, UTC+01:00)
I embeded a windows form control in my web application to do multiple file upload/download using MTOM. It works fine in the windows application. However I always get Unsupported Media Type error. Will MTOM works with this type of control?

<object id="myComponent"
classid="http:multipledownload.dll#[myAplication.test.download]"
height="value" width="value">
</object>
Tao
Tuesday, 15 December 2009 14:45:40 (GMT Standard Time, UTC+00:00)
Hey Tim,

I'm looking for a solution like the one you posted here. However, since your article predates WCF, it doesn't include this technology. Do you have any pointers for this?

Michel.
Tuesday, 15 December 2009 14:49:42 (GMT Standard Time, UTC+00:00)
hi Michael,
i never wrote a WCF version, but someone else did if i remember rightly. there's some discussion on this on the codeproject comments page for this article: here is the search link.
http://www.codeproject.com/script/Forums/Search.aspx?fid=250646&kw=WCF&FrmSrch=Search
best of luck
tim
tim
OpenID
Please login with either your OpenID above, or your details below.
Name
E-mail
Home page

Comment (Some html is allowed: a@href@title, strike) where the @ means "attribute." For example, you can use <a href="" title=""> or <blockquote cite="Scott">.  

[Captcha]Enter the code shown (prevents robots):

Live Comment Preview