WCF ... best practices moving large amounts of data

Hello everyone: I am looking for everyone's thoughts on moving large amounts
(actually, not very large, but large enough that I'm throwing exceptions
using the default configurations).

We're doing a proof-of-concept on WCF whereby we have a Windows form client
and a Server. Our server is a middle-tier that interfaces with our SQL 05
database server.

Using the "netTcpBindings" (using the default config ... no special
adjustments to buffer size, buffer pool size, etc., etc.) we are invoking a
call to our server, which invokes a stored procedure and returns the query
result. At this point we take the rows in the result set and "package" them
into a hashtable, then return the hashtable to the calling client.

Our original exception was a time-out exception, but with some
experimentation we've learned that wasn't the problem .... although it is
getting reported that way. Turns out it was the amount of data.

The query should return ~11,000 records from the database. From our
experimentation we've noticed we can only return 95 of the rows back before
we throw a "exceded buffer size" exception. Using the default values in our
app.config file that size is 65,536 (64Kb).

Not that moving 11,000 records is smart, but to be limited to only 64Kb in a
communication seems overly restrictive. We can change the value from the
default, but I wanted to ask what other's are doing to work with larger
amounts of data with WCF first?

Are you simply "turning up" the size of the buffer size? Some kind of paging
technique? Streams? Some other strategy?? Having a tough time finding
answers on this.

Greatly appreciate any and all comments on this,

Thanks
--
Stay Mobile

Check example called "streaming" on
http://www.idesign.net/idesign/Deskt...dex=5&tabid=11
Arkady

"MobileMan" <MobileMan@discussions.microsoft.com> wrote in message
news:0E54FBD7-5F19-4AF3-823F-AFBD2E6BC84D@microsoft.com...
> Hello everyone: I am looking for everyone's thoughts on moving large
> amounts
> (actually, not very large, but large enough that I'm throwing exceptions
> using the default configurations).
>
> We're doing a proof-of-concept on WCF whereby we have a Windows form
> client
> and a Server. Our server is a middle-tier that interfaces with our SQL 05
> database server.
>
> Using the "netTcpBindings" (using the default config ... no special
> adjustments to buffer size, buffer pool size, etc., etc.) we are invoking
> a
> call to our server, which invokes a stored procedure and returns the query
> result. At this point we take the rows in the result set and "package"
> them
> into a hashtable, then return the hashtable to the calling client.
>
> Our original exception was a time-out exception, but with some
> experimentation we've learned that wasn't the problem .... although it is
> getting reported that way. Turns out it was the amount of data.
>
> The query should return ~11,000 records from the database. From our
> experimentation we've noticed we can only return 95 of the rows back
> before
> we throw a "exceded buffer size" exception. Using the default values in
> our
> app.config file that size is 65,536 (64Kb).
>
> Not that moving 11,000 records is smart, but to be limited to only 64Kb in
> a
> communication seems overly restrictive. We can change the value from the
> default, but I wanted to ask what other's are doing to work with larger
> amounts of data with WCF first?
>
> Are you simply "turning up" the size of the buffer size? Some kind of
> paging
> technique? Streams? Some other strategy?? Having a tough time finding
> answers on this.
>
> Greatly appreciate any and all comments on this,
>
> Thanks
> --
> Stay Mobile

Thanks Arkady. Just received the download e-mail from Juval. Going to check
this out.

Have you used streaming in WCF before? Moved large datasets in WCF? Any
advice based on what I've stated here?


--
Stay Mobile


"Arkady Frenkel" wrote:

> Check example called "streaming" on
> http://www.idesign.net/idesign/Deskt...dex=5&tabid=11
> Arkady
>
> "MobileMan" <MobileMan@discussions.microsoft.com> wrote in message
> news:0E54FBD7-5F19-4AF3-823F-AFBD2E6BC84D@microsoft.com...
> > Hello everyone: I am looking for everyone's thoughts on moving large
> > amounts
> > (actually, not very large, but large enough that I'm throwing exceptions
> > using the default configurations).
> >
> > We're doing a proof-of-concept on WCF whereby we have a Windows form
> > client
> > and a Server. Our server is a middle-tier that interfaces with our SQL 05
> > database server.
> >
> > Using the "netTcpBindings" (using the default config ... no special
> > adjustments to buffer size, buffer pool size, etc., etc.) we are invoking
> > a
> > call to our server, which invokes a stored procedure and returns the query
> > result. At this point we take the rows in the result set and "package"
> > them
> > into a hashtable, then return the hashtable to the calling client.
> >
> > Our original exception was a time-out exception, but with some
> > experimentation we've learned that wasn't the problem .... although it is
> > getting reported that way. Turns out it was the amount of data.
> >
> > The query should return ~11,000 records from the database. From our
> > experimentation we've noticed we can only return 95 of the rows back
> > before
> > we throw a "exceded buffer size" exception. Using the default values in
> > our
> > app.config file that size is 65,536 (64Kb).
> >
> > Not that moving 11,000 records is smart, but to be limited to only 64Kb in
> > a
> > communication seems overly restrictive. We can change the value from the
> > default, but I wanted to ask what other's are doing to work with larger
> > amounts of data with WCF first?
> >
> > Are you simply "turning up" the size of the buffer size? Some kind of
> > paging
> > technique? Streams? Some other strategy?? Having a tough time finding
> > answers on this.
> >
> > Greatly appreciate any and all comments on this,
> >
> > Thanks
> > --
> > Stay Mobile
>
>
>

Interestingly enough, the example experienced the same problem we have run in
to: "{"The maximum message size quota for incoming messages (200000) has
been exceeded. To increase the quota, use the MaxReceivedMessageSize property
on the appropriate binding element."}"

That's the exception that was thrown when attempting to run the long version
of the theme song (if you haven't seen the example, there are a short & long
version of a song that gets played .... one 54KB and the other 184KB). The
app.config was set to a "maxReceivedMessageSize" of "120000" which explains
the problem when playing a file which is 64KB larger.

This runs at the heart of our problem: we'll never know, with any
certainty, what the maximum message size will be between our server and our
client. I guess I can set the value to some obsurd number that we think will
never ocurr ... say, 5GB, but that seems like a less-than-ideal way to handle
this.

Any thoughts?

Thanks
--
Stay Mobile


"Arkady Frenkel" wrote:

> Check example called "streaming" on
> http://www.idesign.net/idesign/Deskt...dex=5&tabid=11
> Arkady
>
> "MobileMan" <MobileMan@discussions.microsoft.com> wrote in message
> news:0E54FBD7-5F19-4AF3-823F-AFBD2E6BC84D@microsoft.com...
> > Hello everyone: I am looking for everyone's thoughts on moving large
> > amounts
> > (actually, not very large, but large enough that I'm throwing exceptions
> > using the default configurations).
> >
> > We're doing a proof-of-concept on WCF whereby we have a Windows form
> > client
> > and a Server. Our server is a middle-tier that interfaces with our SQL 05
> > database server.
> >
> > Using the "netTcpBindings" (using the default config ... no special
> > adjustments to buffer size, buffer pool size, etc., etc.) we are invoking
> > a
> > call to our server, which invokes a stored procedure and returns the query
> > result. At this point we take the rows in the result set and "package"
> > them
> > into a hashtable, then return the hashtable to the calling client.
> >
> > Our original exception was a time-out exception, but with some
> > experimentation we've learned that wasn't the problem .... although it is
> > getting reported that way. Turns out it was the amount of data.
> >
> > The query should return ~11,000 records from the database. From our
> > experimentation we've noticed we can only return 95 of the rows back
> > before
> > we throw a "exceded buffer size" exception. Using the default values in
> > our
> > app.config file that size is 65,536 (64Kb).
> >
> > Not that moving 11,000 records is smart, but to be limited to only 64Kb in
> > a
> > communication seems overly restrictive. We can change the value from the
> > default, but I wanted to ask what other's are doing to work with larger
> > amounts of data with WCF first?
> >
> > Are you simply "turning up" the size of the buffer size? Some kind of
> > paging
> > technique? Streams? Some other strategy?? Having a tough time finding
> > answers on this.
> >
> > Greatly appreciate any and all comments on this,
> >
> > Thanks
> > --
> > Stay Mobile
>
>
>

Well, thinking of the song example, if you are downloading portions of
a file, you could always chunk it - i.e. in the OperationContract
specify the offset and chunk-size (capped) and return (out?) the
number of bytes read (as per the stanadard Stream operations). You
could then make a (limited) number of round-trips to the server
without it blowing up, and without using tons of cache, and allowing
more graceful pickup of incomplete downloads.

Obviously this doesn't work so well if you are actually returning
ad-hoc data from a database. Another suggestion: have you tried
compression? You could use the GZipStream (etc) to compress the data
at the server and re-hydrate it at the client; I believe you can just
return byte[] and the enginer will apply Base64/MTOM as appropriate.
Heck, you might even be able to do it via an extension
behaviour/inspector pair, but I suspect the cache limit would kick in
before it gets as far as "BeforeSendRequest", so this might be a dead
end... just random thoughts really...

Marc

AFAIK you have max of 2 G file size in OS. A lot of WinAPI have that limit
internally , e.g. just try TransmitFile() , as I appreciate that used for
file transmition by WCF. So that's the value can be set for
MaxReceivedMessageSize , unless you want to change it dynamically

Arkady

"MobileMan" <MobileMan@discussions.microsoft.com> wrote in message
news:367D233D-F6AF-4DA1-961D-D5D335AD3EF4@microsoft.com...
> Interestingly enough, the example experienced the same problem we have run
> in
> to: "{"The maximum message size quota for incoming messages (200000) has
> been exceeded. To increase the quota, use the MaxReceivedMessageSize
> property
> on the appropriate binding element."}"
>
> That's the exception that was thrown when attempting to run the long
> version
> of the theme song (if you haven't seen the example, there are a short &
> long
> version of a song that gets played .... one 54KB and the other 184KB).
> The
> app.config was set to a "maxReceivedMessageSize" of "120000" which
> explains
> the problem when playing a file which is 64KB larger.
>
> This runs at the heart of our problem: we'll never know, with any
> certainty, what the maximum message size will be between our server and
> our
> client. I guess I can set the value to some obsurd number that we think
> will
> never ocurr ... say, 5GB, but that seems like a less-than-ideal way to
> handle
> this.
>
> Any thoughts?
>
> Thanks
> --
> Stay Mobile
>
>
> "Arkady Frenkel" wrote:
>
>> Check example called "streaming" on
>> http://www.idesign.net/idesign/Deskt...dex=5&tabid=11
>> Arkady
>>
>> "MobileMan" <MobileMan@discussions.microsoft.com> wrote in message
>> news:0E54FBD7-5F19-4AF3-823F-AFBD2E6BC84D@microsoft.com...
>> > Hello everyone: I am looking for everyone's thoughts on moving large
>> > amounts
>> > (actually, not very large, but large enough that I'm throwing
>> > exceptions
>> > using the default configurations).
>> >
>> > We're doing a proof-of-concept on WCF whereby we have a Windows form
>> > client
>> > and a Server. Our server is a middle-tier that interfaces with our SQL
>> > 05
>> > database server.
>> >
>> > Using the "netTcpBindings" (using the default config ... no special
>> > adjustments to buffer size, buffer pool size, etc., etc.) we are
>> > invoking
>> > a
>> > call to our server, which invokes a stored procedure and returns the
>> > query
>> > result. At this point we take the rows in the result set and "package"
>> > them
>> > into a hashtable, then return the hashtable to the calling client.
>> >
>> > Our original exception was a time-out exception, but with some
>> > experimentation we've learned that wasn't the problem .... although it
>> > is
>> > getting reported that way. Turns out it was the amount of data.
>> >
>> > The query should return ~11,000 records from the database. From our
>> > experimentation we've noticed we can only return 95 of the rows back
>> > before
>> > we throw a "exceded buffer size" exception. Using the default values in
>> > our
>> > app.config file that size is 65,536 (64Kb).
>> >
>> > Not that moving 11,000 records is smart, but to be limited to only 64Kb
>> > in
>> > a
>> > communication seems overly restrictive. We can change the value from
>> > the
>> > default, but I wanted to ask what other's are doing to work with larger
>> > amounts of data with WCF first?
>> >
>> > Are you simply "turning up" the size of the buffer size? Some kind of
>> > paging
>> > technique? Streams? Some other strategy?? Having a tough time
>> > finding
>> > answers on this.
>> >
>> > Greatly appreciate any and all comments on this,
>> >
>> > Thanks
>> > --
>> > Stay Mobile
>>
>>
>>

I agree with your thoughts on chunking. Spent all day yesterday surfing -
reading - surfing - reading .... there's surprisingly little out there on
this subject, which tells me there's not a lot of WCF in production yet. I
have a tough time believing everyone is working with trivial amounts of data.

The few specs of info that I found generally centered around streaming, but
there are a few issues if one decides to move in that direction:

1) Streaming won't support WCF's "Reliable Messaing"
2) Streaming won't support Transactions.
3) Again, streaming won't support WS-Security either.
4) You still have no way of knowing if you have selected an appropriate
MaxReceivedMessageSize so that you don't throw the exception.
5) If you are streaming a large file ... say a 1GB download ... and you
loose your connection (or some other issue that causes a problem), you'll
have to start over again. Admitedly, this really only applies to those
situations where the client requires the entire message ... data from the
database. If the user where streaming a video feed they might be able to
simply forget about what has already been sent, and simply keep moving
forward with the current data??? But, I think you see my point.

I guess point 4 might, or might not, be a big deal. I would assume that the
buffer size that gets defined actually creates a buffer in memory. The point
being if a user specified 500MB (in an attempt to use buffered messaging so
they could enjoy message-level security or transactions, but still cover most
scenarios in returning larger datasets) the program actually consums 500MB of
when creating the buffer. In contrast, hopefully, the MaxReceivedMessageSize
is simply a value that gets checked as the message is being reconstituted???
Can anyone confirm this?

The final analysis: by using some type of chunking routine the client could
stick with a buffered approach (keep security, transactions, reliable
messaging), and hopefully still work with messages of any size.

Frankly, I'm surprised this doesn't exist in WCF already. We do a lot of
mobile communications between servers and handheld devices. Just about all
the communiation code is based on asynchronous socket messaging. We send out
bytes, and read in bytes using a chunking concept. It's extrememly reliable
and fast. I'd rather use WCF in the sense of all the power it has, and
flexibility, but they need a nice bullet-proofed dhunking ability for
buffered communication.

FYI ... found a WCF sample written by Yasser Shohoud (he used to be on MS's
WCF team) whereby he creates a custom "TCP Chunking" binding. Haven't
completed the review yet, and it does have some limitations, but it's a step
in the right direction
(http://wcf.netfx3.com/files/folders/...entry3545.aspx)

Let me know if you have any other ideas, or if there might be a simpler -
better way to handle this?

Thanks
--
Stay Mobile


"Marc Gravell" wrote:

> Well, thinking of the song example, if you are downloading portions of
> a file, you could always chunk it - i.e. in the OperationContract
> specify the offset and chunk-size (capped) and return (out?) the
> number of bytes read (as per the stanadard Stream operations). You
> could then make a (limited) number of round-trips to the server
> without it blowing up, and without using tons of cache, and allowing
> more graceful pickup of incomplete downloads.
>
> Obviously this doesn't work so well if you are actually returning
> ad-hoc data from a database. Another suggestion: have you tried
> compression? You could use the GZipStream (etc) to compress the data
> at the server and re-hydrate it at the client; I believe you can just
> return byte[] and the enginer will apply Base64/MTOM as appropriate.
> Heck, you might even be able to do it via an extension
> behaviour/inspector pair, but I suspect the cache limit would kick in
> before it gets as far as "BeforeSendRequest", so this might be a dead
> end... just random thoughts really...
>
> Marc
>
>
>

Thanks Arkady:

Hopefully you will know the answer to this question. If I stick with a
buffered approach, and take my buffer size attribute and crank it up to ....
say, 1GB, .... I assume this actually would consume 1GB of RAM when the
buffer gets created (on the channel open)?? The point here do I waste
resources by setting the buffer size to an artificially high number, or is
simply something that gets checked as the buffer is getting filled??

I would also assume the MaxReceivedMessageSize attribute does not waste
resources, but is only something that get's checked as the message is being
reconstituted?

Your thoughts on sticking with the buffered approach (check out my comments
to Marc Gravell in this thread), but chunking the received message somehow???
--
Stay Mobile


"Arkady Frenkel" wrote:

> AFAIK you have max of 2 G file size in OS. A lot of WinAPI have that limit
> internally , e.g. just try TransmitFile() , as I appreciate that used for
> file transmition by WCF. So that's the value can be set for
> MaxReceivedMessageSize , unless you want to change it dynamically
>
> Arkady
>
> "MobileMan" <MobileMan@discussions.microsoft.com> wrote in message
> news:367D233D-F6AF-4DA1-961D-D5D335AD3EF4@microsoft.com...
> > Interestingly enough, the example experienced the same problem we have run
> > in
> > to: "{"The maximum message size quota for incoming messages (200000) has
> > been exceeded. To increase the quota, use the MaxReceivedMessageSize
> > property
> > on the appropriate binding element."}"
> >
> > That's the exception that was thrown when attempting to run the long
> > version
> > of the theme song (if you haven't seen the example, there are a short &
> > long
> > version of a song that gets played .... one 54KB and the other 184KB).
> > The
> > app.config was set to a "maxReceivedMessageSize" of "120000" which
> > explains
> > the problem when playing a file which is 64KB larger.
> >
> > This runs at the heart of our problem: we'll never know, with any
> > certainty, what the maximum message size will be between our server and
> > our
> > client. I guess I can set the value to some obsurd number that we think
> > will
> > never ocurr ... say, 5GB, but that seems like a less-than-ideal way to
> > handle
> > this.
> >
> > Any thoughts?
> >
> > Thanks
> > --
> > Stay Mobile
> >
> >
> > "Arkady Frenkel" wrote:
> >
> >> Check example called "streaming" on
> >> http://www.idesign.net/idesign/Deskt...dex=5&tabid=11
> >> Arkady
> >>
> >> "MobileMan" <MobileMan@discussions.microsoft.com> wrote in message
> >> news:0E54FBD7-5F19-4AF3-823F-AFBD2E6BC84D@microsoft.com...
> >> > Hello everyone: I am looking for everyone's thoughts on moving large
> >> > amounts
> >> > (actually, not very large, but large enough that I'm throwing
> >> > exceptions
> >> > using the default configurations).
> >> >
> >> > We're doing a proof-of-concept on WCF whereby we have a Windows form
> >> > client
> >> > and a Server. Our server is a middle-tier that interfaces with our SQL
> >> > 05
> >> > database server.
> >> >
> >> > Using the "netTcpBindings" (using the default config ... no special
> >> > adjustments to buffer size, buffer pool size, etc., etc.) we are
> >> > invoking
> >> > a
> >> > call to our server, which invokes a stored procedure and returns the
> >> > query
> >> > result. At this point we take the rows in the result set and "package"
> >> > them
> >> > into a hashtable, then return the hashtable to the calling client.
> >> >
> >> > Our original exception was a time-out exception, but with some
> >> > experimentation we've learned that wasn't the problem .... although it
> >> > is
> >> > getting reported that way. Turns out it was the amount of data.
> >> >
> >> > The query should return ~11,000 records from the database. From our
> >> > experimentation we've noticed we can only return 95 of the rows back
> >> > before
> >> > we throw a "exceded buffer size" exception. Using the default values in
> >> > our
> >> > app.config file that size is 65,536 (64Kb).
> >> >
> >> > Not that moving 11,000 records is smart, but to be limited to only 64Kb
> >> > in
> >> > a
> >> > communication seems overly restrictive. We can change the value from
> >> > the
> >> > default, but I wanted to ask what other's are doing to work with larger
> >> > amounts of data with WCF first?
> >> >
> >> > Are you simply "turning up" the size of the buffer size? Some kind of
> >> > paging
> >> > technique? Streams? Some other strategy?? Having a tough time
> >> > finding
> >> > answers on this.
> >> >
> >> > Greatly appreciate any and all comments on this,
> >> >
> >> > Thanks
> >> > --
> >> > Stay Mobile
> >>
> >>
> >>
>
>
>

Well, I'm sad to say Yasser's example might have been targeted towards an
earlier version of WCF. We run the RTM vesion that came out back at the end
of November.

Don't know this for sure, but after getting through the compilation errors
(it was trying to override methods that no longer exist) I still ran into
run-time exceptions for similar reasons.

I still feel this is the best approach for reasons I stated earlier, but
we'll have to wait and come back on this (out of time to find an answer).
We'll have to go with streaming for now, but I'm going to come back to this
next month and see if I can't build a "chunking binding" similar to what he
did, but based on the RTM structure.

If you have other ideas, I'd love to hear them,

Thanks
--
Stay Mobile


"Marc Gravell" wrote:

> Well, thinking of the song example, if you are downloading portions of
> a file, you could always chunk it - i.e. in the OperationContract
> specify the offset and chunk-size (capped) and return (out?) the
> number of bytes read (as per the stanadard Stream operations). You
> could then make a (limited) number of round-trips to the server
> without it blowing up, and without using tons of cache, and allowing
> more graceful pickup of incomplete downloads.
>
> Obviously this doesn't work so well if you are actually returning
> ad-hoc data from a database. Another suggestion: have you tried
> compression? You could use the GZipStream (etc) to compress the data
> at the server and re-hydrate it at the client; I believe you can just
> return byte[] and the enginer will apply Base64/MTOM as appropriate.
> Heck, you might even be able to do it via an extension
> behaviour/inspector pair, but I suspect the cache limit would kick in
> before it gets as far as "BeforeSendRequest", so this might be a dead
> end... just random thoughts really...
>
> Marc
>
>
>

Hi, All:

See the site at
http://www.udaparts.com/document/Tut...orialThree.htm for how to move
large size files, large record set, large collection of items and large
whatever by our SocketPro at www.udaparts.com.

We see a lot of semilar problems posted on various discussion groups and
web sites. Let me tell you our SocketPro is able to solve this type of
challenge problems with much more elegant and simpler codes. This tutorial
sample is a good testmony to the quality of our SocketPro. You can also see
our source codes for our remote window file and database services.

We publish this message for helping you and also for advertisement on
internet. Our SocketPro is able to solve many many challenge problems in our
daily programming in unique way, batching, asynchrony and parallel
computation.

Regards,


"MobileMan" <MobileMan@discussions.microsoft.com> wrote in message
news:0E54FBD7-5F19-4AF3-823F-AFBD2E6BC84D@microsoft.com...
> Hello everyone: I am looking for everyone's thoughts on moving large
> amounts
> (actually, not very large, but large enough that I'm throwing exceptions
> using the default configurations).
>
> We're doing a proof-of-concept on WCF whereby we have a Windows form
> client
> and a Server. Our server is a middle-tier that interfaces with our SQL 05
> database server.
>
> Using the "netTcpBindings" (using the default config ... no special
> adjustments to buffer size, buffer pool size, etc., etc.) we are invoking
> a
> call to our server, which invokes a stored procedure and returns the query
> result. At this point we take the rows in the result set and "package"
> them
> into a hashtable, then return the hashtable to the calling client.
>
> Our original exception was a time-out exception, but with some
> experimentation we've learned that wasn't the problem .... although it is
> getting reported that way. Turns out it was the amount of data.
>
> The query should return ~11,000 records from the database. From our
> experimentation we've noticed we can only return 95 of the rows back
> before
> we throw a "exceded buffer size" exception. Using the default values in
> our
> app.config file that size is 65,536 (64Kb).
>
> Not that moving 11,000 records is smart, but to be limited to only 64Kb in
> a
> communication seems overly restrictive. We can change the value from the
> default, but I wanted to ask what other's are doing to work with larger
> amounts of data with WCF first?
>
> Are you simply "turning up" the size of the buffer size? Some kind of
> paging
> technique? Streams? Some other strategy?? Having a tough time finding
> answers on this.
>
> Greatly appreciate any and all comments on this,
>
> Thanks
> --
> Stay Mobile