Friday, December 8, 2017

Validate a JSON Stream before uploading with HttpClient

Today I ran into an interesting challenge that I wanted to record so I can remember it.

I’m building a little service that receives arbitrary files and stores them in S3. However, some of these files will be forwarded to Elastic Search for indexing. Now elastic search is quite lenient with accepting data, even invalid json, and thus will return the invalid json when found, which then causes problems upstream. But, the files can be any size and I don’t want to load the entire stream in memory, then validate it and then forward it.

In C#, you can use a HttpRequestMessage with a StreamContent to send a stream of data to an HTTP endpoint. The problem is, the StreamContent class reads the input stream and copies it to the output stream. My problem was that I wanted to validate the bytes as they were copied from the input stream to the output stream. When sending a HTTP response, in stead of sending a normal SteamContent, I created a PushStreamContent. This exists in the full .net framework, but not in dotnet core.
Then I created a SplittingStream. It takes two streams, an input stream that data will be read from, and an output stream. Any byte read from the input stream will be written to the output stream.

Here’s the result in a gist:
Validate json while sending it over HTTP:


  1. Hi, Great.. Tutorial is just awesome..It is really helpful for a newbie like me.. I am a regular follower of your blog. Really very informative post you shared here. Kindly keep blogging. If anyone wants to become a .Net developer learn from .Net Core Training in Chennai. or learn thru .Net Core Training in Chennai. Nowadays Dot Net has tons of job opportunities on various vertical industry.
    or Es6 Training in Chennai. Nowadays JavaScript has tons of job opportunities on various vertical industry.

  2. A portion of the video streaming programming accessible incorporates 'Macromedia Communication Server', 'Apple Quicktime Streaming Server', 'Helix Universal Server, iptv service 2019

  3. nice article
    very useful information.
    keep sharing.
    best java training in Bengaluru

  4. Design patterns and system design are concepts every professional developers should learn. These design patterns are utilized in developing maintainable and scalable enterprise applications. Thank you for sharing this. Great blog. Data structures and algorithms in Java