last day (23 days later) » 

8:04 PM
4
A: Wikipedia Category Hierarchy from dumps

svickThe category hierarchy information in MediaWiki is stored in the categorylinks table, so you're going to need the categorylinks dump. You're also going to need the page (not pages-articles) dump for page id to title mapping.

 
@fersarr Yeah, that's the one.
@fersarr Yeah, exactly.
 
I'm trying to do the same thing, but maybe the categorylinks schema has changed. It no longer has a pageId column. categorylinks now has [cl_from,cl_to,cl_sortkey,cl_timestamp,cl_sortkey_prefix,cl_collation,cl_type]. How can I build the hierarchy from this?
 
@kane If you look at the documentation for categorylinks, you'll see that the page_id is stored in the cl_from column (and always was).
 
@svick thank you. So categorylinks.cl_from = page.page_id. Where do I go from there? Let's take, for example, page_id=12, which is "Anarchism". There are 19 categorylinks where cl_from=12. The cl_to is a text field [Anarchism, Anti-capitalism,Anti-fascism,...]. How do I find the parent or child categories/pages?
 
@kane Find the page_id for the page with page_title = 'Category:Anarchism' and then look that up in categorylinks etc.
 
8:04 PM
@svick Am I suppose to look up the cl_to values in category.cat_title? The issue is not all of the cl_to values are in category.cat_title. For example, there is a category.cat_title=Anarchism but none for Anti-capitalism. And it just seems odd they didn't list the category.cat_id instead
@svick In my 34M page table, there is not page_title='Category:Anarchism'. There are 9 entries where page_title='Anarchism' however. Do I have the wrong page table, maybe?
@svick In fact, I only have 12 entries in page where page_title LIKE 'Category:%'
 
@kane 34MB? page.sql.gz for the English Wikipedia is 1 GB. Maybe you downloaded it for different wiki? And you don't need the category table at all (notice that I never mentioned it), and especially not the cat_id (it's not related to page_id).
Also, it's not actually page_title = 'Category:Anarchism', it's page_namespace = 14 AND page_title = 'Anarchism'.
(Sorry about that.)
 
hello
ok, let me try to find this 1GB file
I got the enwiki-latest-page.sql.gz (1.21GB) from dumps.wikimedia.org/enwiki/latest
The zip expanded to 3.68GB and it looks like I was only able to load 34M rows (3.4GB) of it. Let me try importing into mysql again
Actually, I think i have all of it. Wikipedia stats say there are about 34M pages. Sorry, didn't mean to say 34MB
And yes, I do see a page entry where page_namespace=14 and page_title='Anarchism'. It's page_id is 780754. categorylinks has 7 entries with cl_from=780754
where do I go from here?
...Do I take each of the 7 cl_to values from categorylinks and then do a search in page where namespace=14 and page_title=cl_to?
So the 7 cl_to's are [Anti-fascism, Political_culture,...].
So, those are the parent categories of "Anarchism".
To find the parent categories of "Anti-fascism", I suppose I do a search in page where namespace=14 and page_title="Anti-fascism"?
I think that's it, unless you tell me otherwise. So thank you very much in advance!
last question: how can I tell which of the 34M pages are articles? Those are the only ones I'm interested in starting from to build my tree
My guess is that the namespace should be 0 according to mediawiki.org/wiki/Manual:Namespace. Can you confirm?
 
8:40 PM
Yeah, you got it all correct.
 
Is there a way to also specify only english articles?
I found 11M articles, but wiki stats say there should 4.6M english articles. How do I filter on english?
 
If you downloaded enwiki-latest-page.sql.gz, that has only English articles
Not sure where could you get that 11M number from.
 
I just queried the page table [select count(*) from page where page_namespace = 0] and I got back "11344969"
 
Maybe you imported it wrong, or something like that? Otherwise I have no idea, that should be the correct query.
 
ok, got it. I'll figure this one out. Thank you again for helping me through this!
 
9:03 PM
you're welcome
 

  last day (23 days later) »