Friday, December 8, 2017

Validate a JSON Stream before uploading with HttpClient

Today I ran into an interesting challenge that I wanted to record so I can remember it.

I’m building a little service that receives arbitrary files and stores them in S3. However, some of these files will be forwarded to Elastic Search for indexing. Now elastic search is quite lenient with accepting data, even invalid json, and thus will return the invalid json when found, which then causes problems upstream. But, the files can be any size and I don’t want to load the entire stream in memory, then validate it and then forward it.

In C#, you can use a HttpRequestMessage with a StreamContent to send a stream of data to an HTTP endpoint. The problem is, the StreamContent class reads the input stream and copies it to the output stream. My problem was that I wanted to validate the bytes as they were copied from the input stream to the output stream. When sending a HTTP response, in stead of sending a normal SteamContent, I created a PushStreamContent. This exists in the full .net framework, but not in dotnet core.
Then I created a SplittingStream. It takes two streams, an input stream that data will be read from, and an output stream. Any byte read from the input stream will be written to the output stream.

Here’s the result in a gist:
Validate json while sending it over HTTP: