Found TCP: Buffer Management and related Async Sockets and Buffer Management and Async Sockets and Buffer Management [CTD] articles on Greg Young’s blog. These articles describe the problem and show the solution:
When you call BeginReceive/BeginSend on a socket, the memory that you pass to the method will be pinned for the duration of the call. This will lead to heap fragmentation and to make matters worse it will often be in your gen0 heap if you aren’t careful. This becomes especially true in the most common socket applications (request/response) as BeginReceive is often called and waits for a long time to receive data (by long time I mean a few seconds or more).
The blog itself is interesting on its own.