last day (15 days later) » 

18:00
0
Q: Upload from app service to azure blob workflow

Amr ElgarhyI have a website running on azure app service, this website allow user to upload a file, after uploading to the server I take the stream and upload it to Azure blob storage. blob.UploadFromStream(stream); The website shows an upload progress percentage bar while the user is uploading the file....

Try reading this article arcware.net/…
@aaronR thank you, just finished reading it, it is good article, but this is somehow exactly what I am doing, he get the file(s) from the user to the server, save it as a local file then upload it back to azure storage so the time spent in this process is (Upload to server time + Upload from Server to blob storage)
So what type of client do you have?
@aaronR what do you mean by client? the user do you mean?
Is is web browser or native mobile app?
18:00
It is a website
Are the customers the general public or are they corporate in nature where they would have a good internet connection? Will the users be uploading files from their smart phones?
different type of users and yes they may upload from their phones, it is a public website, so I have no control of the users types.
So you are really limited by the users internet/cellular connection speed. What you could do is put the file upload logic into Azure functions so they won’t impact your other server functions/speed. I would do some do some baseline testing first on your configuration before trying to optimize.
But isn’t the azure functions way will be the same work flow? Which is upload the file to the function then the function upload it back to the blob storage
It is but will scale and will allow your users to upload files without impacting each other or your main functions of your web app. And you only pay for what you use and first 1 million calls are free.
Make sure your web app has compression enabled too
18:00
Yes, you are right, all of these are very important and will improve my upload performance but will not affect the time of uploading from my app service to the blob storage, get me?
Hit me up on Twitter so we can talk further to get you a solution then share it with others. @cajunAA
We can chat here as well
hey ther
hey
thank you for chatting here
have you calculated some metrics to see how long it takes to upload some files from the web client to the azure storage and the segments, like time from client to web api and from api to azure storage?
no prob
18:08
I have done some logging for that
ok great
does the web api wait for the entire file before saving it to Azure Blob?
a file 178MB from my computer to the website server take takes about 5 minutes, then from the server to the blob storage takes exactly 8sec
Yes
It wait for the entire file till it receive the file
is the client "chunking" the file to the web api
actually my controller is action mvc not web api
ok are you .net core?
18:11
nope
ok its conceptually the same then even pre .net core the controllers are almost the same base logic
Yes, exactly
you could look at using the blob method of saving the chunks at a time to the blob
the problem is that my controller action doesn't start going inside it's code until the whole file received
the PutBlock
18:15
so this is my controller action: public ActionResult SubmitFiles(HttpPostedFileBase file)
then the code inside this action method doesn't fire fire till the whole file get received to the server
then I take the file and call this:blob.UploadFromStream(stream);
ok we need to change that to accept chunks of the file. the client may have to change to
that will help to incase of a network drop or issue during the upload
do you mean to change the SubmitFile Action method?
yes im looking up now
tyt, you are a star
this article covers the concept
it uses a HTML5 control, not sure I would use that
but it could work as a prototype to see if it speeds up the transfer
when you have file chunking enabled you may be able to do some optimization on the client to start more than one connection to the MVC controller to upload different parts of the file
18:25
yes, i saw this article before, but this HTML5 will skip the server, the file will go directly to the blob from the browser, in my case I can't do that because I have some logic happen on the file before uploading to the blob such as renaming the file and prepare the path it will go to in the blob based on many factors
when you say chunking, do you mean uploading from the browser to my app service or from the app service to the blob storage?
chunking on both segments
so send pieces to the web MVC controller and that in turn takes the chunks and saves to Azure blob
You know, if I managed to let the client post the file in chunks to my app service this will a great
Yes, exactly
this will be great
I will try to do that, need to do some tests, not sure yet if possible
how many users do you expect to be uploading files at the same time?
on peak times max 100 users
but they upload big files
around 1 GB sometimes
ok you will want to get the code to work first with your MVC controller
18:29
Yes
then create an Azure Function that does the same thing to handle that
exactly
try this article on the JavaScript side of chunking
thank you for giving my direction and ideas to search in
no prob
18:30
yes, interesting
I will update it here with what I will get
I had to implement picture uploads from mobile APP and I had to write the API
my users are on cellular and uploading lots
so we just let them know they may want to wait to upload until they were connected over WiFi
make sense
I read up on these optimization but didnt implement them yet so I need this info anyway lol
here is the first article in the series
18:33
:)
hit me up on twitter if you need anything else
sure, many thanks for your time
you are welcome

last day (15 days later) »