last day (14 days later) » 

5:25 AM
1
A: How do I case fold a string in Python 2?

RussCheck out py2casefold. >>> from py2casefold import casefold >>> print casefold(u"tschüß") tschüss >>> casefold(u"ΣίσυφοςfiÆ") == casefold(u"ΣΊΣΥΦΟσFIæ") == u"σίσυφοσfiæ" True

 
J.F. Sebastian -- what would you add? For reference, here is the extent of python 3's str.casefold unit test.
 
see whether you'll manage to fish out a useful test from regex:test_case_folding()
 
J.F. Sebastian -- Thanks for the link, but there wasn't much useful I could see in that regex testing (a comment or two would be nice in there!). Even still, I wasn't super pleased with the slim unit testing either, so I beefed it up a tad. There really isn't much to the casefolding operation, though. Not a heck of a lot that can be tested.
 
I can't believe that something is uncomplicated in Unicode e.g., are you sure no casefolding properties have been changed between different versions of the Unicode standard -- may I expect that casefold works the same between different Python 2 versions? Read what @tchrist says on the related topic
 
5:25 AM
J.F. Sebastian -- I know what you mean, but case folding is actually fundamentally pretty simple. @tchrist is 100% correct, but he is basically just saying to use the unicode casefold "algorithm" and not lowercase. As far as functionality between python 2 versions, operation should be 100% identical. The case folding operation is basically just a lookup table. The key is to use the master unicode table, which is currently at 8.0.0 (and included in the package).
Oh... and all the @tchrist tests will definitely pass (if he got them right :) ). Several cases are already covered, and are pretty standard for case folding.
howdy
 
@Russ I see the difference between py3casefold, py2casefold, and icu.UnicodeString.foldCase
 
meaning you see "a" difference, or you see that there is a difference?
py2casefold shoudl match the python 3 implementation
aside from being slower, of course
 
e.g., `print(u'%04X %s -> %s vs. %s -> %s' % (ord(c), c, casefold(c), c, py3casefolded))`
-> `ABAA ꮪ -> Ꮪ vs. ꮪ -> ꮪ`
 
the icu one might not have the latest casefold data, also. My local Qt 5.5, for example, is stuck on 7.0.0, and 5. is the latest/greatest.
parsing that
 
the above is for py2casefold vs. stdlib str.casefold in Python 3
 
5:31 AM
python3 may have an older one, too... depends on when the unicode grabber tool was run. Probably not relevant here, though...
what is your test string, exactly?
nevermind... ABAA
getting late here... one sec and let me compare
Interesting... python3 is wrong (but probably just out of date??).
It is leaving "\uABAA" alone, but the unicode code folding standard clearly says to change it to "\u13DA", even with the common folding case.
backchecking in v7...
J.F. Sebastian -- yep, v7 did not have it, your python3 (and ideone's) are just out of date with unicode is all.
 
docker run -it --rm python python -V -> Python 3.4.3
 
Clearly showing this:
 
where do I find the Unicode version for a given python executable?
 
last I checked python3 it had a unicode data fetching tool... let me check
not sure if they stash the version in use for a build. The fetching tool makes a lookup table, though... they do not use the file directly.
This is the tool:
ABAA; C; 13DA; # CHEROKEE SMALL LETTER DU
How did you come up with the 0xABAA example, out of curiousity? Did you ramp through chars?
There were 150 new folds in v8 vs v7, with 1350 total in v8. That you picked one of those out of the sea of unicode codepoints makes it seem like you systematically looked.
 
5:58 AM
found Unicode version: `unicodedata.unidata_version` -> 6.3.0

docker run -it --rm python python -c 'import unicodedata as U; print(U.unidata_version)'

I've tested all individual Unicode codepoints, in Python 3:

json.dump({c: c.casefold() for c in map(chr, range(sys.maxunicode + 1))},
sys.stdout)
 
I am not 100% certain if that unidata_version is for the CodeFolding.txt file. Are you? I don't see it in the cpython source.
unicodedata has all the normalizations, names, etc... unless the CodeFolding.txt file was lumped in there at the same time? I have no idea.
I'm starting to think that Python3's str.casefold is quite out of date.
And everywhere I say "CodeFolding.txt" above I meant "CaseFolding.txt"... not sure what my brain/fingers were doing.
 
Moreover my brain read it as CaseFolding.txt :)
 
:)
need to sleep now, but I'm still quite certain that py2casefold is 100% correct, based on the latest unicode data.
Also want to dig out the actual age of the py3 casefold map... could be inferred if I dig out the CaseFolding.txt history and look at where python3 gets it wrong vs py2casefold.
Anyway - thanks for bashing on py2codefold ... I was quite sure it was robust, but am more sure now. :)
Good night!
 
 
2 hours later…
7:52 AM
I've tested on Python 3 with custom unicodedata.unidata_version == '8.0.0' and py2casefold produces the same values for all individual Unicode codepoints -- it is disorienting that py2casefold uses CaseFolding.txt 8.0.0 while python 2 where it runs may use much older version such as unicodedata.unidata_version == '5.2.0'
 

last day (14 days later) »