last day (17 days later) » 

12:46
0
Q: Error Bad state: Stream has already been listened to

NaveenI've been trying to chunk the file and upload it to an AWS presigned URL. For the very first iteration, it works, but after that, I'm getting the error Error in _getChunkStream: Bad state: Stream has already been listened to I've tried different approaches, but nothing worked. I've been stuck on ...

check ChunkedStreamReader - the docs say: "Utility class for reading elements from a chunked stream."
but another question is: do you really need to use multiple uploads? is it a limitation of your server that does not accept files with big sizes?
@pskink What is happening is, I'm creating a multipart request for S3. It will give an upload ID, and after that, I'll get presigned URLs. I'll upload the files to those URLs. For example, if my file size is 200MB and the chunk size is 50MB, I'll get 4 presigned URLs, and I'll upload 50MB to each of them. Once that is done, I call a CompleteMultipart API, which will merge all the URLs and make it a final file.
Here are the relevant AWS documentation links for reference: - CreateMultipartUpload - UploadPart - CompleteMultipartUpload
@pskink I tried this package .But still i'm getting the same bad state pub.dev/packages/chunked_stream Stream<List<int>> _getChunkStream(int start, int end, int chunkSize) async* { print("reading from $start to $end"); try { final reader = cc.asChunkedStream(end, file.readStream!); yield* reader .asyncExpand((chunk) => Stream<List<int>>.fromIterable(chunk)); } catch (e) { print("Error in getchunked stream $e"); } }
and where is the upload limit size specified?
_maxChunkSize = min(fileSize, maxChunkSize);
I mean where is it specified that you have to use multiple uploads to send one file
Ok I see it now
It's CompleteMultipartUpload
so as I said you can simply use ChunkedStreamReader class
13:06
so what is unclear with ChunkedStreamReader?
I've added chunkedStreamReader. But still same issue
Stream<List<int>> _getChunkStream(int start, int end, int chunkSize) async* {
    print("reading from $start to $end");

      final r = ChunkedStreamReader(file.readStream!);
    try {

      yield* r.readStream(end);

      // final reader = cc.asChunkedStream(end, file.readStream!);

      // yield* reader
      //     .asyncExpand((chunk) => Stream<List<int>>.fromIterable(chunk));
    } catch (e) {
      print("Error in getchunked stream $e");
final fakeData = utf8.encode('total €19');
print('fakeData: $fakeData');
final fakeDataLength = fakeData.length;
const maxChunkSize = 4;
final fakeStream = Stream.value(fakeData);

final reader = ChunkedStreamReader(fakeStream);
int chunk = 0;
final client = http.Client();
for (int i = 0; i < fakeDataLength; i += maxChunkSize) {
final chunkSize = min(fakeDataLength - i, maxChunkSize);
print('<< sending chunk $chunk, chunkSize: $chunkSize');
final request = http.MultipartRequest('post', Uri.parse('http://localhost:8000/'))
it sends fake data (a short string actually) divided in 4 bytes chunks: just replace final fakeStream = Stream.value(fakeData); with your stream
Ok am i doing anything wrong in _getChunkStream ?
you call ChunkedStreamReader multiple times since you call _getChunkStream multiple times
in the code i posted ChunkedStreamReader is called once even thou multiple chunks are sent
ok can i try increasing maxchunksize to 1024 * 1024 * 50 ? from your code?
13:19
sure
is 50 MB a server limit?
i mean per one upload
nope
so can i send the whole file? some files might be anywhere around 1 - 2 gb
so why you want us use that limit?
why not 50 GB for example?
I tried to send 512 mb before. but my ui got stuck
i dont see any difference if in my code you use maxChunkSize = 50MB or maxChunkSize = 50GB
try 50 GB and check it out
13:34
can i call this code in a loop? i need to assign them to particular s3 link
what loop do you mean?
inside a for loop
yes, why not?
for (var part in parts) {

          uploadRequest.url = part["signedUrl"];


          uploadRequest.fakeData();

        }

 Future<void> fakeData() async {
    final fakeData = utf8.encode('total €19');
    print('fakeData: $fakeData');
    final fakeDataLength = fakeData.length;
    const maxChunkSize = 1024 * 1024 * 50;
    final fakeStream = file.readStream!;

    final reader = ChunkedStreamReader(fakeStream);
    int chunk = 0;
    final client = http.Client();
    for (int i = 0; i < fakeDataLength; i += maxChunkSize) {
fakeDataLength is still wrong
final fakeDataLength = fakeData.length;
13:52
Future<void> fakeData() async {
try {
final fakeDataLength = file.size;
const maxChunkSize = 1024 * 1024 * 50;
final fakeStream = file.readStream!;

final reader = ChunkedStreamReader(fakeStream);
int chunk = 0;
final client = http.Client();
for (int i = 0; i < fakeDataLength; i += maxChunkSize) {
final chunkSize = min(fakeDataLength - i, maxChunkSize);
print('<< sending chunk $chunk, chunkSize: $chunkSize');
final request = http.MultipartRequest('put', Uri.parse(url))
..files.add(http.MultipartFile(
file.readStream is from file_picker
this my file_picker code
static Future<PlatformFile?> pickLargeFile({
    List<String> allowedExtension = const ["zip"],
    bool withReadStream = true,
    FileType fileType = FileType.custom,
    dynamic Function(FilePickerStatus)? onFileLoading,
  }) async {
    try {
      FilePickerResult? result = await FilePicker.platform.pickFiles(
          type: fileType,
          allowedExtensions: allowedExtension,
          withReadStream: withReadStream,
          onFileLoading: onFileLoading);
      if (result != null) {
is final reader = ChunkedStreamReader(fakeStream); called only one time with the same stream?
yeah it was called multiple times. now i've changed it. but nw i'm getting Bad state: Concurrent read operations are not allowed!
where are those Concurrent read operations?
Future<void> fakeData(ChunkedStreamReader<int> reader) async {
    try {
      final fakeDataLength = file.size;
      const maxChunkSize = 1024 * 1024 * 50;

      int chunk = 0;
      final client = http.Client();
      for (int i = 0; i < fakeDataLength; i += maxChunkSize) {
        final chunkSize = min(fakeDataLength - i, maxChunkSize);
        print('<< sending chunk $chunk, chunkSize: $chunkSize');
        final request = http.MultipartRequest('put', Uri.parse(url))
          ..files.add(http.MultipartFile(
what are the logs?
do you see "<< sending chunk $chunk, chunkSize: $chunkSize'" and ">> received chunk $chunk, statusCode: ${response.statusCode}"?
14:07
<< sending chunk 0, chunkSize: 52428800
>> received chunk 0, statusCode: 200
Response headers : {content-length: 0, etag: "de9e09aa25b0b74734df426f6770f75d"}

<< sending chunk 1, chunkSize: 52428800
>> received chunk 1, statusCode: 200
Response headers : {content-length: 0, etag: "f47ab0ecdd5d91cf984609884c8e8e1f"}
<< sending chunk 2, chunkSize: 16147
>> received chunk 2, statusCode: 200
Response headers : {content-length: 0, etag: "fe0435099b042f24a7edc14aac67d015"}
i've uploaded 100mb file
52428800+52428800+16147 is it a file size you want to upload?
i mean 104873747
is it the file size?
Yes
so the 3 chunks were uploaded correctly
whats the problem then?
but it should be 2 right? coz my chunk size is 50 mb i've uploaded 100 mb
can i ignore this error Error in fake data : Bad state: Concurrent read operations are not allowed!?
no 104873747 / 2 = 52436873.5
52428800 * 2 = 104857600
104857600 is not 104873747
14:12
yeah got it
i have no idea what it cames from since you have 3 uploads with status code 200
what about the concurrent read operations? can i ignore it?
i have no idea what it comes from since you have 3 uploads with status code 200
which means all of them were succesful
Let me try merging all the files
14:37
is it a good approach to set the maxchunksize to 2gb?
that won't work ig coz for 512 mb file i got RangeError: Array buffer allocation failed
where is that RangeError?
what file and what line number?

  last day (17 days later) »