« first day (1356 days earlier)      last day (1876 days later) » 

3:04 AM
posted on March 18, 2019 by Steve Eddins

I was noodling around recenctly with a future blog topic (multiresolution pyramids; stay tuned) when I was reminded of a recent addition to MATLAB: the function imtile, introduced in R2018b.... read more >>

 
 
8 hours later…
10:55 AM
@AndrasDeak I am starting to suspect i tmay have to do with lower level malloc() and free() libraries.... So lost
its there, if you have a CUDA capable GPU and CUDA installed
 
nope
but at least I might get a better idea of what the code is like
I've got a nice integrated gpu :P
 
Today between segfaults I got "Error in python:double free or corruption (!prev)" and this lead me to tensorflow bug reports that claim it has to do with the library that does malloc() D:
@AndrasDeak "nice"
 
it's never given me any trouble :'D
That link is an entire branch...I suspect there's more to the problem than that? Some entry point at least, when you're executing it and get a segfault?
 
Yes yes. It errors when running detest.py in TIGRE/Python. The code that errors (after being executed, the internal code runs) is in Python/tigre/Source/_Ax.pyx
 
thanks
 
11:01 AM
detest.py lol nice function name
 
Verb: detest (third-person singular simple present detests, present participle detesting, simple past and past participle detested)
  1. (transitive) To dislike intensely; to loathe.
  2. I detest snakes.Alexander Pope
  3. Who dares think one thing, and another tell, / My heart detests him as the gates of hell.
  4. (transitive, obsolete) To witness against; to denounce; to condemn.
  5. Fuller
(3 more not shown…)
 
Interestingly, its very similar to Python/tigre/Source/_Atb.pyx, but this last one does never errors
 
@Dev-iL exactly :D
 
D:
devtest.py, in fact XD
 
lol
Ander strikes again
 
11:02 AM
I just wrote it wrong XD
 
I'll search for "tyop" in the commit history later...
 
hahahahaha
 
from libc.stdlib cimport malloc, free
    # Numpy must be initialized. When using numpy from C or Cython you must
    # _always_ do that, or you will have segfaults
 
> Who dares think one thing, and another tell,
My heart detests him as the gates of hell.
 
I'm curious; does the order of imports not matter?
 
11:04 AM
that's a good one ^
 
@Dev-iL do you detest Ander for thinking devtest and saying detest?
 
@AndrasDeak I do not know. I am not an expert I did nto write this code.
 
@AndrasDeak Maybe he also thinks it wrong... how do you know?
 
The comment you added though is for the next line
 
@AnderBiguri one difference between _Atb and _Ax is the order of imports. I'd try using the same order as the former
@AnderBiguri oooh I see
but that's also one difference, actually
 
11:06 AM
hummmmmmmmmmmmmmmmmmmmm
I will test that
 
in _Atb you have the init inside _Atb_ext
dunno which one makes sense, but if you have segfaults and a comment warning against segfaults, I'd try investigating these differences
 
urf, I'm reading the README on someone else's FORTRAN code, and half of it is in French, with stuff like reels*8 instead of reals*8. Hurts my eyes
 
@AndrasDeak yeah you are right ah. It did not ocurr me to look there, I just recently started thinking that it may be due to malloc, and not a bug in the code. Im on it
 
@Adriaan both are wrong, actually
@AnderBiguri well I told you that if the numpy copy finishes it should be safe afterward :P
 
real*8? Probably, sorry
 
11:11 AM
yup
 
@AndrasDeak but it is not, or something is not. the behavior is so unpredictable that I do not know anymore. But calling the Ax function without using the output will never error.
Anyway, the import order is not the cause
 
@Adriaan ugh, ancient fortran naming conventions
@AnderBiguri probably because nothing is trying to access the freed memory so it goes below the radar
@AnderBiguri and moving the init inside the function just like with _Atb?
 
@AndrasDeak that is technically wrong, it shoudl be outside. Aparently is not needed always, so I am not sure if its needed in my case
 
11:17 AM
@AndrasDeak the files themselves have a function-description, but there's no README which tells me which file I need to look at :(
 
@Adriaan grep -i "what_you_need" *.f
 
but yeah, it doesnt work either
Again, its super strange, because I get segfault after I successfully proj.sum() the output, and returns the correct value
 
because if it's freed the data might still be there
 
Why is that different to imshow() tho
 
As I said I don't think there's anything special about imshow, other than perhaps it trying to get nontrivial stuff about the array. Shape, flags, stuff like that.
my point is that if you have freed memory and UB then it's an unstable mess
 
11:26 AM
yes, but I dont understand why I have freed memory
I should not
 
So...are you certain that # Free pointer array, not actual data doesn't actually mess with the underlying data?
might be worth a temporary memory leak to check that
I don't think it should mess with it, but again, you have two indirections there and two frees where in the other non-segfault function you just have the one
Also, if you're going to return projections.swapaxes(1,2).copy(order='C') then why not create that copy first, and free the original arrays later? Also, shouldn't you free the memory of the data before returning, since you don't need the original array and you have a copy?
(which is to say you might have a memory leak in the current version)
 
@AndrasDeak not in C, definetly
@AndrasDeak copy needs to go
its there temporarily, until I change the row/column major stuff
 
OK
segfault makes even less sense to be honest
 
11:42 AM
@AndrasDeak quadruple checked, removing that does not change anything
 
thanks
 
same with the copy
same behaviour of random segfaults or memory-related errors
 
weird
 
such as the one I posted earlier, or "Error in python: corrupted size vs prev_size"
 
Hmm, I'm wondering...this malloc and free is not python malloc but system malloc, right?
 
11:47 AM
yes
I think
 
yeah
> Note that the C-API functions for allocating memory on the Python heap are generally preferred over the low-level C functions above as the memory they provide is actually accounted for in Python’s internal memory management system. They also have special optimisations for smaller memory blocks, which speeds up their allocation by avoiding costly operating system calls.
 
I see
 
but I do need ownership of the memory, as CUDA will do stuff with it, such as pinning it
 
this shouldn't be the issue though; system malloc means that you have to free yourself, so it's more error-prone on the leaky side
I don't really know what could be off :/
 
11:52 AM
This can be
I will investigate
In MATLAB, when the mex exits, the process gets killed and the memory is freed by the OS. Not sure about python.
but I don't think so
 
system malloc will only be freed on process exit
python malloc would probably do some refcounting and whatnot
 
yeah, but in MATLAB, mex files are a separate process
And my students have literally translated more than I wished for
as in, instead of finding the pyhton way, they jus trasnlated line by line
despite my insistence of them not doing it so
 
ugh
 
huh compiler dislikes PyMem_Free(c_projections)
 
and not the PyMem_Alloc?
 
11:58 AM
no
But I am getting here into bigger trouble
 
hehe
 
because the compiler errors because I try to grab pointers to the variable now
 
as I said, I don't think your problem would be fixed by this
 
but this I want to do
I am thouhg using PyArray_SimpleFromData that shoudl create the numpy cvariables from data
and the next line with the flags should give ownership to Python, so it can free
 
yeah, but that's numpy C API, not python C API
As far as I can tell what you have should work. You allocate memory, set it, create an array that is a wrapper to that data and tell it that it also owns the data. When the array dies it should free the underlying memory. But the array should only die when the function returns, by which point the copy is already created...
Does cython have the equivalent of -C or --check bounds?
 
12:08 PM
yeah.... And the CUDA code can be an evil bastard, but the exact same CUDA code runs in my MATLAB version....
 
wonder if it might happen that you're overwriting something else in memory due to an indexing bug, and that's what's causing nonlocal issues
 
@AndrasDeak no idea what those are
 
sorry, I meant compile flags that do array bounds checking
oh, that's probably not a thing in C, shoot
in fortran arrays have well-defined sizes...
 
@AndrasDeak thought about that, but I have looked a lot. And the code runs on MATLAB. And on Python, sometimes, always with the same input
 
well I don't know how deterministic handling of virtual memory is
if you index 15 bytes outside the stack of this function, do you always end up in the same place?
 
12:10 PM
mmm wait why is virtual memory coming to the topic?
 
Dunno how anything works :D
Programs aren't one big chunk in memory, are they?
 
I mean, I am not using virtual memory in theory, all is properly allocated memory in RAM
 
Ignore the vitual
 
so what is the question?
 
Will independent parts of the code always occupy the same relative position in memory across separate runs?
 
12:14 PM
I still do not get it
each of these calls should be independent. Creates memory, fills it,gives it to python
 
like "main() starts at 0x3acf23f49234, so other_function()'s stack always starts 58249 bytes to the right"
 
I have no idea how to check that nor why I shoudl be worried about it
 
@AnderBiguri hmm, true, I guess I'm again thinking in static memory
 
doesnt the compiler/python handle all that?
 
it does
@AnderBiguri but if memory gets reshuffled relatively, then an out-of-bounds write will overwrite different parts of your code on different runs, hence potential for Heisenbug
 
12:16 PM
yep yep
Thus there is somethign dodgy with how memory is handled
 
I'm only trying to argue whether it's a technical possibility that an out-of-bounds index leads to non-reproducible issues
 
but that I suspect already
@AndrasDeak in my experience, this segfaults all the time
 
8 mins ago, by Ander Biguri
@AndrasDeak thought about that, but I have looked a lot. And the code runs on MATLAB. And on Python, sometimes, always with the same input
 
the instant you do Array[size+1]
 
so what's "sometimes"?
 
12:17 PM
you get a segfault
 
@AnderBiguri oooh, I see
 
@AndrasDeak sometimes runs
 
yeah, I misunderstood "this" in that message
 
ah, I shoudl be clearer sorry
anyway, going to have a lucnhbreak see if I can clear my mind and thing of something else
 
yeah, sure
I'm having lunch as we speak
did pudb not lead to anything?
 
1:06 PM
@AndrasDeak not really
 
survey: do you guys use ipermute?
 
@Dev-iL no
 
interesting fucntion
neveer used it, but I should
 
looks useful indeed
 
@AnderBiguri My thoughts exactly
 
1:18 PM
I don't think I've ever needed it
 
To my understanding, it's for the cases you want to permute a matrix, perform some operation on it, then permute it back to what it was before, w/o having to rethink what the correct order of dimensions is
 
yeah, I don't ever do that
 
some of us do :)
 
1:35 PM
-1
Q: Matlab Trapz giving weird results

existence is futileI am trying to find the area under the curve however I got very weird result. In the first picture, potentiostat calculates the peak area even higher than the total area of the trapz calculation. I tried abs() but no change, I could not find what is wrong here.

I think OP did not realize that a*b=c, c does not need to be bigger than a and b
 
unclear
 
Has anyone seen the term 'non-existant' before? All I get from interwebz is 'common misspelling of nonexistent'; this paper I am reading from a couple of native speakers (Australians) uses it often in for terms in a recursion formula which are not (yet) available
Oh, and a bunch of results in French
 
Sam
1:53 PM
o/
@Adriaan not existing or not real or present.
 
@Sam so sort-of a misspelling still :P
thanks
 
when in doubt, wiktionary
the non-negated also only gives the common misspelling, in English
had to undo the one-box because it was confusing
 
Ya, just realised, the rest is about French
 
shitty one-box
 
Hey guys
 
1:56 PM
sup, snippy
 
@Ad all your family okay and everything?
 
is that Ad for Andras Deak or Ad for Adriaan? :P
 
@ballBreaker iirc none are in Utrecht
 
oooh, shit's down in Utrecht?
 
Shoulda been for Adriaan, but hopefully your family is okay too :D
 
1:57 PM
shooting, damn
 
Shooting incident; possible terrorist attack. 1 dead, several severely injured.
 
Sam
Shooting? Where?
 
Utrecht ^
 
Yeah it's where Connie did her exchange, I'm pretty surprised. It seems like a pretty random and peaceful place
but I guess none of this really makes sense anymore
 
peaceful place or not, you only need one nutjob
 
1:59 PM
@ballBreaker fourth biggest city in the country; and apparently the suspect lived like 100m from where the shooting took place
 
Sam
@AndrasDeak preach.
 
Who here knows their way around VS?
 
Sam
@Dev-iL I've dabbled
 
@AndrasDeak very true
@Adriaan jeeze
That sounds less terrorist, and more like the dude just snapped
 
distinction's getting a bit blurry these days
 
2:01 PM
I downloaded some source files, now these files should be compiled in 3 separate "groups" (projects?), so that they don't interfere with each other
 
lot of things are still unclear; I'm following the live blog/stream more than doing actual work atm
 
how do I set this up?
 
@AndrasDeak also very true
 
the first group is "header1, code1, code2" the 2nd group is "header1, header2, code3, code4" etc
right not the project contains all headers and all source files - which is not good
 
I take it "new project" and moving stuff is difficult/not an option?
 
Sam
2:03 PM
Can you manually separate them into their own projects?
 
I guess I could, but I prefer to keep all of them in the same folder
 
do you know for a fact that that's possible to begin with?
 
nope
 
Sam
@Dev-iL Doesn't seem a usual approach
More prone to errors when building the solution having completely separate code bases I'd imagine
 
well that's what they do on linux apparently... the makefile says: that object consists of these files, and the other object uses the other files
 
Sam
2:06 PM
You could try hacking the xml(?) make file which tells the compiler which headers and source files to include in the solution
 
Bleh... I thought this was something trivial to do
 
Sam
do you have a flat folder structure? i.e. header1, header2 is at the same level in the directory?
 
yeah
 
Sam
hmm no my idea wont work thinking about it
@AndrasDeak is this a clean way of returning only a single element from a list comprehension?
 
@Dev-iL I'd argue that's the same project but with multiple build targets
 
Sam
2:12 PM
next(iter([key for key, value in self._searched_store.items() if value == child]))
 
Well the list comp will be created in its entirety, so it's simpler to get [0] from it. But you could swap the listcomp for a genexp and then you don't need iter. That's what I'd do.
and you can set a default in the next() call to handle an empty generator, which is not possible when indexing a listcomp
 
Sam
@AndrasDeak I'm not familiar with genexp?
 
next(key for key, value in self._searched_store.items() if value == child)
generator expression, nowadays officially called a generator comprehension
it's the lazy equivalent of a listcomp
 
Sam
ahhh ok. Would this genexp only work if wrapped in a next()?
 
no the generator object itself is (key for key, value in self._searched_store.items() if value == child) which you can iterate over, or take items from it with next(), or call list() on it to consume it, etc. In other words, it's a lazy iterator.
 
Sam
2:18 PM
nice. thanks
 
no problem
 
3:00 PM
In case anyone is interested in computing spherical harmonics ranging from orders 10^-5000 to 10^5000 this article is a very good read
 
 
2 hours later…
5:04 PM
posted on March 18, 2019 by Cleve Moler

I recently acquired a GPU, a graphics processing unit. It's called a GPU because such processors were originally intended to speed up graphics. But MATLAB uses it to speed up computation. Let's see how the gpuArray object benchmarks on my machine.... read more >>

 
 
1 hour later…
6:08 PM
So much moniez ^
 
I wont comment on it, in case SO calls me a hater again :P
 
6:27 PM
@AnderBiguri hater
 
 
2 hours later…
8:54 PM
Sigh.
Nice. You just disproved my comment. Apparently this is how Stack Overflow works. Dump your homework question and someone will answer it for you. You robbed OP of a chance to learn, and you made it harder for all of us to moderate this website. Thank you! — Cris Luengo 53 secs ago
Too rough on the new guy? :)
@Dev-iL interesting. Now I won’t have to compute the inverse manually.
 
@Dev-iL Apparently only once: stackoverflow.com/questions/38336017/…
@CrisLuengo Given my frustration level with some of the questions today, I am compelled to say no, not too rough. :/
 
9:19 PM
@Dev-iL yer a rich man!
 
9:36 PM
@LuisMendo @flawr you seen this? github.com/mame/quine-relay
2
@CrisLuengo nope
 
@AndrasDeak Would be impressed if someone came with this on the resumee:)
 
10:07 PM
Apparently my comment was too harsh, it got deleted. :/
 
11:00 PM
yeah, it was already deleted when I replied
That doesn't mean that it was too harsh, only that it was flagged. Practically any comment will be deleted when flagged.
 

« first day (1356 days earlier)      last day (1876 days later) »