« first day (1973 days earlier)      last day (1238 days later) » 

2:41 AM
How can I get rid of this warning in MATLAB and show the remaining output?
"Warning: For increased performance remaining outputs are not shown. Consider reducing the number of outputs"
In my live script.
 
 
7 hours later…
9:27 AM
@AndrasDeak what series are you chatting about?
 
9:56 AM
@AndréYuhai have you considered reducing the number of outputs?
 
10:30 AM
@AnderBiguri I just wanted to save them all in the same PDF so that's why I didn't reduce anything. :D Instead I changed my function to make it converge faster haha.
 
well, the second option is the much better one, indeed :)
 
10:59 AM
Hi all!
 
@flawr The Mandalorian
 
Did any of you every try writing to a log file from within a parfor? It appears that things are only written when the actual parfor iteration ends... Any idea if this can be forced?
 
@Dev-iL hey
 
I end up with log files looking like this:
[23-11-2020 20:49:22.983] Starting work on: I0092_AT02_AP072_B1_F4.mat
[24-11-2020 06:28:45.702] Iteration #92 took 34762.7174
[23-11-2020 23:12:38.844] Starting work on: I0233_AT02_AP083_B2_F4.mat
[24-11-2020 08:15:56.057] Iteration #233 took 32597.2092
[23-11-2020 19:14:23.561] Starting work on: I0127_AT02_AP107_B1_F4.mat
[24-11-2020 08:16:09.224] Iteration #127 took 46905.661
[23-11-2020 23:12:18.697] Starting work on: I0213_AT02_AP063_B2_F4.mat
[24-11-2020 08:19:50.498] Iteration #213 took 32851.7955
 
I don't think you should
i.e. I/O is the slowest part of coding, over hundreds of times sower than computation. So parallel computing strategies don't generally compute AND save to file
that said, I don't think what you are seeing is what you think you are seeing, as parfor won't necesarily execute trheads in order
[24-11-2020 08:19:50.498] Iteration #213 took 32851.7955
[24-11-2020 08:33:40.065] Iteration #72 took 43342.2362

As you can see above, #213 was called before #72
 
11:14 AM
yeah that's completely uninteresting
the issue is that I need to run like 3k cases, and for this I need to submit a bunch of jobs to a cluster. At the moment I have no way of getting different processes to "not step on each other's toes" other than checking that a result file with a specific unique name already exists
the problem is that I cannot "reserve the spot" by creating the file at the beginning of the loop, because all side effects (like file I/O and log outputs) seem to only happen at the end of iterations
 
so you only want to run process #X if process #Y has finished?
 
I want to run like 150 workers and they need to pick "available" jobs
so that no 2 workers compute the same thing simultaneously
and they can't really communicate (perhaps if I switch to spmd they could)
 
I see. Wouldnt it make more sense to have 1 job per worker
 
that's one of the possible solutions
 
is there any reason not to do that?
 
11:27 AM
At the moment I lack the bash skills needed to create all those jobs
 
@AnderBiguri capitalism
 
hahaha, hum, maybe
@Dev-iL what about just running in catches of 150?
batches
If you know how to call 150 (or nworker) jobs, then you can just make a for loop in bash, its relatively easy
(its bash, meaning its klingon, but that should be doable)
 
@AndrasDeak Yes, at last! :-D
 
@AnderBiguri might as well do a 3k loop
 
I am thinking I may have spoiled stuff in my previous messages, for people who have not watched the series. Sorry!
 
11:31 AM
ah, yeah, dunno how your system works. My local clusters can only let me queue X workers at a time with a minimum, so I would need to say book 150, then 150, etc
 
@AnderBiguri we probably have some limitation in place, never reached it though
 
@LuisMendo the only thing that worries me about that series is how the storm troopers alwasy miss all shots, except when the shoot mando, then they always hit him in the armor.
@Dev-iL 3K sounds like above the limitation 😃
 
@AnderBiguri Yeah... have you reached episode 8? (This doesn't really spoil anything). They shoot at a can for fun and miss all the time, haha
 
hahah dont remember that. I think I am on par, I have watched up to season 2 episode 4
 
I couldn't even get the damn bash script to accept an input.. It worked for a bit, but then stopped working for an inexplicable reason
(and when I say "inexplicable" I mean "explicable" in the sense that I know the reason but not how to solve it)
 
11:39 AM
@Dev-iL I hate bash, don't really know what could be the problem
I think its one of those thigns that its only alive because people who worked on computers oin the 70s are still around and refuse to learn python :D
 
so basically when you call a .sh file, any space-separated strings after the file name act as positional inputs you can refer to with $1...$N
 
@AnderBiguri Here
 
hahaha I remember now yes 😆
 
however, when I submit the work to the cluster, I do not call the .sh directly, but provide it as an input to qsub... and I found no way to specify inputs that the .sh could understand
 
@AnderBiguri normally you can submit whatever you want but at most N will be running at a time
 
11:43 AM
@AndrasDeak depends on the cluster, these things are generally super personalized
 
@AnderBiguri yeah, but your cluster is dumb
 
ah, I was assuming devil here was submiting to some uni cluster or something like that
 
@Dev-iL I'd definitely try to run one per job for what it's worth
let the scheduler worry about scheduling
 
@AndrasDeak I came across that q&a, not sure I tried all the solutions, but those I did didn't work
@AnderBiguri I am
 
@Dev-iL if qsub corresponds to SGE then accoding to the man page -v should work
So your issue may have been elsewhere. Might be worth checking with a test job whether variables get set
 
11:48 AM
I tried various variations of qsub [ options ] [ command | -- [ command_args ]]
I'll give named variables a go
 
the Q&A I linked to specifically suggests -v
the two top answers at least
 
 
4 hours later…
3:52 PM
@LuisMendo And in the new trilogy, one storm trooper takes off his mask and becomes an expert marksman. I think those masks are not good for aiming.
 
4:22 PM
@CrisLuengo Maybe they are only meant to scare people :-D
 
typical low-quality military gear
 
 
2 hours later…
6:32 PM
This question is interesting. Hard to give an answer, though
It may depend on things such as computer architecture and Matlab version. It is likely related to JIT, but JIT changes across versions, is not documented, and seems to be deactivated by profile, which is precisely what you would use to look into this...
 
yeah, exactly my thoughts
maybe the JIT in the non-vectorized loop does a better job at using CPU-level isntructions to read and add coalescent memory together, while the vectorized version just first fills some matrices, then does the operation
but this is just a guess that I can not really answer
 
Pity that this sort of low-level access is almost impossible to look into with Matlab
Or maybe it's better that way :-D
 
but JIT being disabled, that I did not know. Thats baaad
 
Yeah :-( It limits the usefulness of profile
Anyway, that information is from 2011. It may have changed
I just posted these ideas as a (second) comment to that question, in case they can help
 

« first day (1973 days earlier)      last day (1238 days later) »