last day (14 days later) » 

3:10 PM
1
A: numpy/scipy build adjacency matrix from weighted edgelist

igavriilTry something like the following: import numpy as np import scipy.sparse as sps A = np.array([[0, 1, 1],[0, 2, 1],[1, 2, 1],[1, 0, 1],[2, 1, 4]]) i, j, weight = A[:,0], A[:,1], A[:,2] # find the dimension of the square matrix dim = max(len(set(i)), len(set(j))) B = sps.lil_matrix((dim, dim)) ...

 
Thanks for the tip! I thought there was some built-in function inside numpy and/or scipy to do that...
 
I think that there isn't one. You may want to consider using networkx for creating and manipulating graphs.
 
I tried it, but the problem is that the network is huge: around 150 million nodes. NetworkX crashes because of its dimensions. Your script is good, but it gets a MemoryError when it builds the B matrix. :-(
 
What if you replace B with a sparse matrix?
 
Do you mean inizializing a sparse B matrix? The code now crashes here: B = np.zeros((dim, dim)). But I'm running the script on a machine with 200 GB of RAM, I thought that a np.array was less memory addicted...
 
3:10 PM
Something like this import scipy.sparse as sps ... B = sps.lil_matrix((dim, dim))
did the sparse array made it work?
 
Thanks, let me run again now the script!
 
maybe @unutbu solution is more efficient as he is not looping through the elements and constructing the array at glance.
 
I'm running now the @unutbu script, let's see...
 
I didn't expect that networkx couldn't handle this(especially with 200GB Ram)
 
Yes unfortunately adding edges to the network in networkx is crazy...so that I want to perform a DFS algorithm within scipy to get a subgraph to later load into it...
with the new sparse matrix everything seems fine so far...let's see if it crashes building the todense() matrix for the DFS function...
:-( the coo.todense() is running out of memory...
 
3:22 PM
why do you want to convert it to dense?
maybe you can work your breadth first search with the sparse array..
this will slow your performance (i think) but will save you memory
 
yes I was thinking of it
Thanks for your support! At least now I can read the whole network in memory very fast without killing machines memory!
 
16
Q: What scalability issues are associated with NetworkX?

conradleeI'm interested in network analysis on large networks with millions of nodes and tens of millions of edges. I want to be able to do things like parse networks from many formats, find connected components, detect communities, and run centrality measures like PageRank. I am attracted to NetworkX b...

check also this question. You may want to consider(hopefully not) more hardcore solutions. Good luck with this!Jason
 
Thanks a lot!!
 

last day (14 days later) »