« first day (2502 days earlier)      last day (2449 days later) » 

10:02 PM
or use .ravel() to view it as 1d
 
user8451312
After i turn that to array, i have a piece of code like this:

X_train=[re.sub(r'http\S*', "URL", tweet) for tweet in X_train]
X_train=[re.sub(r'@NAXIStaxi', "mentionnegative", tweet) for tweet in X_train]
X_train=[re.sub(r'@banca_intesa|@TelenorBanka|@RaiffeisenBanka|@kombank|@TelenorSrbija', "mentionpositive", tweet) for tweet in X_train]
X_train=[re.sub(r'[0-9]', "", tweet) for tweet in X_train]
X_train=[re.sub(r'_', " ", tweet) for tweet in X_train]
Because i wrote it before for post_text, instead of X_train, which is the same thing, just the post_text is a list of strings, and this is
 
so how did X_train become a multidimensional array when it used to be 1d?
does train_test_split convert the dimensionality of your inputs?
 
user8451312
X_train shape is (3000, 1) and post _text shape is (5377, )
 
You were worried about runtimes earlier if I remember correctly. So I suggest writing a function that does all those substitutions one after the other, and iterate over X_train only once
@hope94 so X_train is a column-shaped 2d array. In other words, you can trivially convert it to a 1d array without loss of information
so perhaps do something like this:
old_shape = X_train.shape
X_train = [big_sub_function(tweet) for tweet in X_train.ravel()]
X_train = np.array(X_train).reshape(old_shape)
where big_sub_function(tweet) does all those substitutions
 
I have a list of Objects, I want to wrap those Objects with my custom Object, and to return the list again. Till now I was using listcomp, but it's very slow when your list len is 1000+. How can I do it faster ?
 
10:08 PM
but when your pattern is not actually a regex, you can also use the built-in string .replace method
@SpazaM you mean you have a list of objects?
 
@SpazaM [CustomObject(obj) for obj in my_list]
 
ah, now I understand
 
Edited
 
listcomp is as good as you get in general, I think
your Object needs to lose some weight perhaps
 
Faster than a listcomp... don't think so. Even if there's some faster alternative, that sounds like optimizing in the wrong place.
 
10:10 PM
It's only getting more weight
 
user8451312
andras, i don't really understand that code :(
 
user8451312
i just have many different things that i did with post _text to do with x train, its not only about these piece of code that does substitution...
 
@SpazaM - a 1000-element list in and of itself is really nothing to Python. Does the __init__ of CustomObject do a lot of work? If so, perhaps that can be deferred until whatever it is is actually required.
 
@PaulMcG It's inherit from BaseObject and just assigning given args to private vars.
 
user8451312
@AndrasDeak Maybe i find another way to split data?
 
10:18 PM
@PaulMcG It's init doesn't doing any "calculate" job
 
@hope94 sorry, those few messages were directed at SpazaM
@hope94 if all those operations are element-wise, then work with X_train.ravel(), and in the last step reshape it back to a 2d column array if necessary
if it doesn't have to be a 2d column array, just .ravel() it right after creation
you need to understand the tools you're using in order to tell if you need a 2d array or not
(fixed it)
 
@SpazaM - does "inherit from BaseObject" mean that its __init__ calls super().__init__?
 
@PaulMcG I so hope that there's a hidden framework somewhere that is actually called BaseObject :)
 
user8451312
@AndrasDeak that will work i think, thank you so much Andras!
 
OK, good luck with it, no worries
rhubarb
 
10:24 PM
@PaulMcG yep
 
Does BaseObject.__init__ do a lot of work? I mean, the time has to be going somewhere
 
@AndrasDeak lol, it's called BaseVirtualMachine
@PaulMcG that Base also assgin the given args to two private vars
@PaulMcG Do you tell me that running over a list with 1000 Objects, and wrapping them with another Object, using listcomp, need to take few seconds ?
 
If that
 
No, doing nothing significant for a not significant number of objects shouldn't take a significant amount of time
The listcomp isn't the culprit here. There must be something else that's eating your performance
 
class Base:
    def __init__(self, a, b):
        self.a = a
        self.b = b

class CustomWrapper(Base):
    def __init__(self, other):
        super().__init__(other.a, other.b)
        self.other = other

class Z:
    def __init__(self, a, b):
        self.a = a
        self.b = b
With those classes, I created a list of 1000 Z's, and then wrapped them in CustomWrapper, and this was about 0.015 seconds
 
10:32 PM
Python 3 ?
 
But of course
But Py2 would be similar
 
I use 2.7.13, I saw that there is a small change with liscomps and genexp in py3
They are faster, but how much faster ?
 
The difference is probably far smaller than any application logic in your script will notice
 
@PaulMcG Something to notice, the CustomWrapper is like 1000 lines of code, it has a lot of functions, and properties.
 
May I recommend the cProfile module so that we don't have to guess where the problem is?
'Cause I don't think we're heading anywhere
 
10:36 PM
Try testing your list comp with something very very very simple first, then add to it to see where things get bad
 
@Rawing Goos idea, short explanation what should I expect to see ?
@PaulMcG Will try to do.
 
Hi

I am trying to lock an image on my laptop
fd = open('/home/John/Desktop/camera.jpg', 'r')
try:
    fcntl.lockf(fd, fcntl.LOCK_EX)
    print('lock successfully acquired')
except IOError, e:
    print('smth went wrong when acquiring the lock')
It always prints "smth went wrong when acquiring the lock"
am I doing smth wrong in this tiny piece of code? Yes, the path is correct.
 
Try printing e for a clue
 
@PaulMcG [Errno 9] Bad file descriptor
 
Default open with mode of 'r' is text, for a jpeg you probably want 'rb'
 
10:40 PM
@PaulMcG rb didn t solve the issue, error missage is still the same
 
@SpazaM Something like this I guess:
>>> cProfile.run('[i for i in range(10000)]', sort='cumtime')
         3 function calls in 0.002 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.002    0.002    0.002    0.002 <string>:1(<module>)
        1    0.000    0.000    0.000    0.000 {range}
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}
 
@LandonZeKepitelOfGreytBritn - going to the docs: "On at least some systems, LOCK_EX can only be used if the file descriptor refers to a file opened for writing."
 
@PaulMcG what solution do I have if I want to lock an image in order to read it?
I am doing multiprocessing ie writing the file to my hdd using some c++ code and reading it with a python script. I have to somehow lock the image in order to not obtain corrupted data
 
@Rawing thank you, will try
 
I've not done overt file locking for a while - you could copy to a temp file of your own, annd then read that perhaps?
 
10:47 PM
@PaulMcG won't work, the image is originated from a video stream. I am writing every x-th frame of the stream to my hdd. WHich means this image gets overwritten every now and then
The python script needs to obtain a lock between two write operations from the C++ process
in order to be able to obtain correct results
 
@LandonZeKepitelOfGreytBritn open doesn't return a file descriptor. See docs.python.org/3.6/library/os.html#file-descriptor-operations
 
@PM2Ring - he is using Py2, but the issue is the same.
 
@PaulMcG indeed py2
os.open(path, flags, mode=0o777, *, dir_fd=None)

Open the file path and set various flags according to flags and possibly its mode according to mode. When computing mode, the current umask value is first masked out. **Return the file descriptor for the newly opened file**
apparently the open function does return the file descriptor
 
@LandonZeKepitelOfGreytBritn: That's os.open.
 
The built-in open returns a File Object, which has a fileno() method to get the corresponding fd
 
11:00 PM
os.open is a different function to the built-in open function
Sorry I'm a bit slow, I'm on my phone.
 
It may be easier to use the built-in open like you are already doing (like jpeg_file = open('whatever', 'rb')) and then pass jpeg_file.fileno() to fcntl.lockf.
I changed the name from fd since we have determined that the built-in version of open does not really return an fd.
@LandonZeKepitelOfGreytBritn are you still there?
 
@LandonZeKepitelOfGreytBritn Oh. OK. So will the C++ program wait when it attempts to write to the locked file?
 
sorry left 2 seconds
@PM2Ring yes it should go as follows: 1) cpp writes image to hdd while locking it 2) cpp code unlocks image 3) python code reads image and locks it 4) python unlocks 5) go to step 1
 
PM2Ring's question is more like "What does cpp do if at step 1, the Python code still has the lock from the previous step 3?"
 
@PaulMcG like so: fcntl.lockf( jpeg_file.fileno() , fcntl.LOCK_EX) ?
@PaulMcG wait to write
 
11:15 PM
Yes, something like that
 
it should be some sort of 1 single intraprocess mutex
 
But you'll just have to experiment in the Python interpreter, whether you need to open with mode r, w, rb, or wb and so forth. But I think the biggest stumbling block was that the builtin open gives not a file descriptor but a file object, from which you can get the fd.
Be sure to do your Python lock/release using try-finally (release the lock in a finally clause) so that, in the event of an exception, the file will still get unlocked
 
@PaulMcG I still got bad file descriptor:
 
Did you try all four modes, r, w, rb, and wb?
 
if __name__ =='__main__':

    jpeg_file = open('/home/John/Desktop/camera.jpg', 'rb')
    try:
        fcntl.lockf( jpeg_file.fileno() , fcntl.LOCK_EX)
        #fcntl.lockf(fd, fcntl.LOCK_EX)
        print('lock successfully acquired')
    except IOError, e:
        print(e)
        print('smth went wrong when acquiring the lock')
@PaulMcG w and wb worked. But actually I am tryin to read the file not write to it or smth
because what happens now is that
I had a genuine .jpg image in my folder, by using w or wb as a parameter the image got deleted and replaced by a file called camera.jpg
when using r or rb as parameters this doesn t happen, but I get a bad file descriptor exception
 
11:23 PM
You might try opening with 'r+b', which opens for both read and write
But should not overwrite the existing file
 
Hmm. Looks like Python or the OS doesn't want to lock read mode files. And a plain write mode open always truncates the file. So you'll have to use one of the read-write modes.
 
@PM2Ring nice that worked :)
now I ll try to test it for intraprocess locking
... once I finished my cigaret
 
/me losing interest (I nominate "okra" as the salad vocab for this condition)
okra
 
Well, it looks like we solved the 2 core problems.
 
Seems like locking would be useful to add to the pathlib.Path class. And it also seems like a natural for a context manager.
We have some legacy code at work that uses fcntl, I'll take a fresh look at it tomorrow
Dinner's on - rbrb!
 
11:42 PM
To check the existence of a file, Shall I use .is_file()?
 
user8451312
I have a problem with this part of code. It runs infinitely long.
I don't know where is the problem. I tried tu run custom_tokenizer(X_train) and it works fine. But when this is run in count vectorizer, its so slow and never finishes.

    #define tokenizer
    def custom_tokenizer(text):
        #split- space
         tokens = ' '.join(X_train).split()
         tokens = [token for token in tokens if len(token)>=3]
    ##     stem tokens
         for i, token in enumerate(tokens):
             for key, val in d.items():
 
@hope94 Sorry, you killed me with the last two lines...
 
user8451312
@EnderLook Why is that? And i am sorry if i did.
 
@EnderLook I assume that you've already googled your question and found this, right?
 
there must still be something wrong somewhere...
C++:
while(1)
{
int fd = open("/home/John/Desktop/camera.jpg", O_RDWR |O_CREAT, 0666);
int rc = flock(fd, LOCK_EX | LOCK_NB);

if(rc==0)
{
qDebug()<<"OK obtained lock";
cv::imwrite("/home/John/Desktop/camera.jpg", imgFrame);
}
else
{qDebug()<<"didnt receive lock: "<<rc;}

}

this code returns: OK obtained lock - didnt receive lock- didnt receive lock- didnt receive lock- didnt receive lock- didnt receive lock, ....
which means the file is still locked, but the python script is still able to access the image without any problem while the cpp c
 
11:47 PM
@vaultah Yes, I found that...
@hope94 I don't know what means the last two lines.
 
@EnderLook what exactly are you asking then?
 
user8451312
@EnderLook Well if you don't know what count vectorizer is you can't help me.
 
@vaultah There are 4 ways to check that. I am not sure which use.
@hope94 I'm sorry.
 

« first day (2502 days earlier)      last day (2449 days later) »