last day (15 days later) » 
04:00 - 08:0008:00 - 10:00

4:39 AM
0
Q: Different Variants of UTF-8 Comma? [,] [,] - CURL Response for MySQL Data

Brian BrumanPrepping a Curl Response for particular data to be inserted into a MySQL Table. Noticed some special characters in the saved data for certain URL's. $curldata = curl_exec($curl); $encoding = mb_detect_encoding($curldata); brought back ASCII encoding. Okay, don't want that. The tables in my ...

 
I agree, but that is seemingly only how it appears. The browser, MySQL, PHP, so forth, all claim it to be the same character (a simple comma) image link
 
online-toolz.com/tools/text-unicode-entities-convertor.php says ,, is %uFF0C%2C, i.e. %uFF0C and a simple comma.
 
Yeah you got me thinking the same. Made a second edit... based off this answer: stackoverflow.com/questions/395832/… ... confusing how the browser interprets it as the same character though. Now trying to figure out how to prevent non-standard UTF-8 characters in my responses
 
The curl requested site is also controlled by yourself? Or do you just try to replace some characters from a foreign response?
 
No, it's an external site. Not exactly sure the ideal solution but I'm looking for maybe a function that will 'normalize' the string to standard utf-8 characters before my PDO/MySQL insert... hm
 
4:39 AM
Isn't there any curl response header providing the actual encoding? Or a <meta> tag?
 
<meta charset="utf-8"/> on the page and Content-Type: text/html; charset=utf-8 in the headers.
 
However $encoding = mb_detect_encoding($curldata); says ASCII
I had issues with auto detection in the past
 
Yeah I was trying to reproduce that. It's strange. For the same URL sometimes it will say ASCII, other times UTF8.
 
depending on the content
You might have more success with php.net/manual/en/function.recode-string.php
The command line tool recode also supports actions like utf8..flat..utf8
I guess the PHP function as well.
However, the PHP function is not well documented. But you can look for more information on the web.
 
I'll give it a shot. Was trying to see if there was some other solution... googling "uff0c" php, "U+FF0C" php, etc.. not bringing back many results. Maybe I'm being too particular. I know I could remove it or add a space and only allow normal characters, perhaps relying on PHP to know a FULLWIDTH Comma should be a normal Comma (or any other instance of a close variant) might be too much to ask without a very verbose function. Or maybe I'm being way too particular.
I'll try yes
 
5:06 AM
I might have a solution...
 
Yes?
Oh I see. Testing now :)
 
I hope it can help without side-effects
 
5:25 AM
Yes! Awesome man!! Thank you. That does indeed work as intended in testing. Is soo obscure and weird haha.. had me googling all over the place.
How in the world did you know to use "a" (Convert "zen-kaku" alphabets and numbers to "han-kaku")
 
Oh, I asked google about converting half/full widh characters and came over this function
and just tried if it is applicable for western characters as well
A really interesting question!
 
For sure, didn't think there would be an answer. Had no idea such a function existed. Thanks for the effort I learned some things along the way. Will have to fine-tune it and do some more testing.
 
After we found out together what the characters actually are, I got an idea how to look for the appropriate keywords.
I like the seemingly impossible things :)
What I dislike is coding the always same things again and again, one day in red, the next day in green.
That's the annoying part of the business.
 
5:40 AM
Figuring out the seemingly impossible things! Creativity and problem solving is where the real magic happens. Also hate monotony. That's why this question intrigued me.
Not implying that converting a full width comma to a normal comma is magical, but it's close considering there's very little to go on to find a solution :)
I'm still really confused. Checking out examples hotexamples.com/examples/-/-/mb_convert_kana/…
You can combine the types... $text = mb_convert_kana($text, 'asKV', 'UTF-8'); seems fitting. Maybe.
"zen-kaku" to "han-kaku", a space variant.. "han-kaku kata-kana" to "zen-kaku kata-kana"... V is "Collapse voiced sound notation and convert them into a character. Use with "K","H""
Really really confused but I like it :)
 
Since there is no good official documentation, there could be some try'n'error method needed. I would use the default (internal encoding) to process the data.
that's usually utf-8 too
 
Yeah, true. I was totally misdirected when I first asked, now it at least makes sense.
The character is on the page, for some reason, not just the curl response.
 
I've never had the idea converting between char widths could become an issue.
 
Never even really knew it was a thing. Assumed there was some way to mass-normalize it, but apparently my notion of what a UTF-8 character was lacking
Welp, I'll keep chipping away at the stone :) Much appreciated again.
 
Yes, foreign data is always a problem. I had to juggle with different CSVs writing an import into database.
You never knew what the next CSV file would break. Horrible!
UTF, Unicode? Oh that's not easy. The first bits tell something about the byte length. It's a science on itself.
I got into the basics, however, I guess there's much more to know I do not even expect
As in ASCII, there's a single bit distinguishing between lower/upper case. I assume in unicode there are much more similar systems to help to convert between different character forms
 
5:58 AM
Just looking at the table and all the variations that a unicode can have for different processors, and it's uniformity is a bit bewildering as well..
I found another solution by the way, without mb_convert_kana
$str = preg_replace_callback(
"/[\x{ff01}-\x{ff5e}]/u",
function($c) {
// convert UTF-8 sequence to ordinal value
$code = ((ord($c[0][0])&0xf)<<12)|((ord($c[0][1])&0x3f)<<6)|(ord($c[0][2])&0x3f);
return chr($code-0xffe0);
},
$str);
Not really sure how it works, but it does :)
1
Q: how to transform japanese english character to normal english character?

SuHun HanI have an japanese english character. This character is not normal english string. Characters: Game How to transform this character to normal english character in php?

Well I guess "Subtract 65248 from the ordinal value of each character."
is how it works.. and yes, converts to a normal comma
 
It starts with a regular expression just replacing a character
better said a character range
then the callback checks the bits by bitwise AND
as long as mb_ function works properly I would use that. Native code should perform much faster
 
Yeah :) I've never used ord() before, or really dealt with binary data or data at it's core.. it is really interesting stuff.
 
not a check, but a bitwise operation.
 
Maybe a back-up function
 
ord should give the byte at a string position
the filtered character range seems to be a 3-byte utf code
if you know how utf is constructed, the bit operation is not the big issue. i like bitwise ops since they are pretty direct and fast
even if the say in c++ the compiler optimizes everything, i really like to build algorithm on bitwise operations.
and from perl i used to handle cryptic code :D
 
6:12 AM
So it takes like 0xEF 0xBC 0x8C which is the UTF8 HEX code for the full-width comma
returns it to its single part
0x2C
What is the significance of 65248, do you know?
 
What do you mean by "significance"?
 
That's brilliant man. I'm just getting started. Have been learning on my own for only 2 and a half years. Just getting into OOP stuff. It's been a journey for sure.
Apparently subtracting that number from the characters returns its normal utf-8 case in that instance
If i'm understanding properly.. if I'm half-understanding properly :)
 
decimal numbers are not really "significant" at all. you have to investigate the bit pattern
hex numbers are near to bit patterns, each digit = 4 bits
0xF=15=0b1111
 
I see. I'm looking into this too, crazy stuff. This is getting down near the rabbit hole, how thing's work on the fundamental level.. I haven't been brazen enough to take a full look yet
 
when you are in the computer science, you only think in bits
there's nothing than 1/0, true/false
 
6:22 AM
You're scaring me
I'm just kidding :)
 
oh it's really cool
 
Really is fascinating.. you've sparked some new pathways for sure.
 
some complicated problem can often be done so easy in just changing some bit with and/or/xor/not
e.g. you have 4 persons. now check all combinations who could go to a party: no one, all, person a and b, person a, c and d ...
you could build weird array combinations in nested loops
 
Really yeah. And looping through the possibilities, the answer emerges.. or a close approximation.
 
or just count from 0 to 15 and you have each combination, each bit representing one person whether he is gooing to the party or not
 
6:26 AM
That makes it seem simple, yes
Fascinated by automation.. haven't gotten into algorithms yet, but building the blocks and learning from the data.. its just like unbelievably powerful, just on the surface it seems
I have a long way to go and the more I learn the more I realize how much I don't know
 
with some bit operation and/not/or/xor you can check condition like: when A goes, B does not. when C goes D goes as well...
yeah you should lern bitwise operation and always think: could i solve this problem bitwise?
 
Ha, I was just reading into it and was going to ask you if by bit operations you mean bitwise operations
Right right
 
if there is no idea, then you can think about loops aso.
yes bitwise, 110 and 011 is 010 - only second bit is "and"
then you can shift bits to get it into position for some operation
about AND/OR you can think of masks
 
masks?
 
variable AND 1100 means clear all bits that are 0 in the mask
variable OR 1100 means set all bits that are 1 in the mask
the mask is 1100 here
NOT inverts, so NOT 1100 is 0011
this is used e.g. in IP/netmask
 
6:38 AM
It's never been so clear... also never so distant. Binary I've always had a backseat fascination for, but never had a reason to learn or apply it because it seems.. well I've never had a practical reason to use it, yet.
 
192.168.1.1/24 has the netmask 24 bits, i.e. 255.255.255.0 or hex FFFFFF00
that's binary 11111111.11111111.11111111.00000000
so the full first 3 octets (bytes) are the net, the remaining last octet is the host
network 192.168.1.0 host 1
the net is just IP AND MASK
host is IP AND NOT MASK
 
Didn't know that..
hm
 
not 11111111.11111111.11111111.00000000 = 00000000.00000000.00000000.11111111
 
I'm just like staring at it lol. I'm getting it, not fully, but I'm getting it.
Thanks for blowing my head off a bit.. is that a pun i don't know
 
AND means clear all bits that are 0 in the mask, so the last octet remains as host
 
6:44 AM
Should really get back to work on my project though.. appreciate the insight.
 
Yeah, I hope I could inspire you to get deeper into bit operations
 
Why octets? Is that just for IPs?
Or..
 
one octet is a byte, i.e. 8 bits
since there are 4 byte in an IP, we say octets
 
Damn, I should know this. That's really good.
Yeah.
 
there can be other masks like 30 bits
meaning 30 bits 1 from the start, then 2 bits as host
 
6:47 AM
How long have you been at this? Did you start with perl?
 
ok, that's unusual since there is a network and broadcast address. only 2 is a little bit small
 
I can tell you're thinking in bits :)
 
i started with c64 basic, then assembler on c64
on pc i started with turbopascal, later delphi and wrote the first web cgi application in turbopascal :D
then i came to perl
those days all web apps were CGI, there was no mod-php or not even fast-cgi
and javascript 1.2 was horrible slow and more for making senseless jokes
 
That's truly awesome.
 
today JS has become one of the fastest languages
 
6:52 AM
Do you do many self-projects? Projects of your own conjuring, your own ideas? I'm curious if the ideas or creative process have stagnated over time.
With your experience.. you would think all you need is the idea, am I wrong?
 
oh yeah, i've problems with content and creative design
never give me a brush into hands :D
 
That's what got me in. There's a lot of gripe about PHP but it's a good starter language, I think.. my main motivation is web-based development, not software, thus far anyways.
Yeah, I hear you there... but the data, man
 
i need creative heads doing the design
 
Design is a whole nother battleground yeah
 
and i avoid learning new things
 
6:55 AM
yeah I got that when you mentioned you enjoy figuring out the impossible
 
a few years ago, my friend - he is 3d artist - came and said, i need to develop 3d applications
 
yes?
 
i did not like the idea. then he called, oh, there's a project, we can earn money. ok, i learned unity3d/C#
 
Was it successful?
 
next time he called, i have to come to a company. we need to solve a project on hololens. i went there as an expert for hololens performance optimization, however, i never heard about the hololens at all :D
yes it was
we also managed the hololens project
they did not realize that we just learned it the first time, lol
you know the microsoft goggle?
 
7:00 AM
I was fascinated by hololens ever since I heard about it years ago
Goggle?
No
 
goggles
glasses
how do you say in English?
 
Google or Goggle? lol
Google Glasses?
Augmented Reality
 
yes AR
as far as i know, google should be goggle. it was a typo
similar as rammstein, do you know the band?
should be ramstein, the city of their first performance
 
Before it was even cool man, before any of the first iteration of VR stuff, AR has always been my fascination. A dream of tech becoming so powerful and small you can fit it into a contact lens
 
but they did not correct the typo
 
7:03 AM
Yes I know of them
Really
Are you from europe
 
i was fascinated of Elite, one of the first 3d games even on c64
vector graphics. simple but the start of 3d rendering
yes, europe
germany
and you?
 
States, Michigan
 
if you said you were german, i would beat the head on the table, lol
 
I am german
 
but it's ok, nowaday i'm used to write in english
 
7:07 AM
I just live in the states :)
 
nicht wahr oder?
 
I know none of my relatives.. nor the language, just my heritage, just my ancestry
All I've got is the DNA
 
i think most us people do have german roots rather than english
 
My mother and my fathers side of the family, as far back as we can go, both lived mainly in germany, a few other scattered european countries
 
as far as i know they missed only 1 vote, otherwise german had become the official language
 
7:10 AM
All that is lost for me though
Who is they?
 
not sure if the citizens were asked, i guess no, but the gremium when founding the early us
usa
many people migrated from germany, it was the dream
the dream of a better life
now all is the same bullshit
NATO, war everywhere
anti-terror laws
freedom is only fiction anymore
we will lose cash money, have to work like slaves...
 
But, technology
 
yeah, best used for observation :D
look at china
 
I am an idealist when it comes to it, I see both sides, but technological innovation is opening up more pathways, more opportunities, more of a voice, more potential than could ever have even been conceived of in the past
It is the revolution, if we can keep it in our hands, it is anti-establishment, it is anti-ideological
The great goal of technology and communication and connectivity I think is to relieve us of our separateness
 
there are cameras everywhere in the public recognizing faces. they have an account of 1000 points. it decrements when they walk over red trafficlights, meet the wrong person, ... when the points are low they cannot by flight or train tickets...
 
7:21 AM
It is hard to vision right now, but it is possible I believe
It's terrifying, yes
 
yes encryption is a nice technology. i'ld like to learn more
however, i expect there already are first quantum computers at NSA
at least they will be soon
 
I think they already have them.
Google has some, even
 
yes. i don't know how many bits they are managing
indeed there are prototypes with few bits. they still are error prone
 
Corporations are becoming more and more at odds.. or I suppose there is much collusion, but corporations are overtaking the government in a fundamental way
 
but that's in the public science
 
7:25 AM
There are some compatibility issues I hear (joking)
What if augmented reality reaches a point where ignorance is no longer an option?
 
when they have a bigger quantum bit size, 256-4096, common encryption is obsolete
you mean AR with no goggles?
just the illusion with your own eyes
 
What if virtual reality reaches a point where you can literally experience another reality in its full essence, with full empathy
Not illusion
Making the invisible visible
 
when you cannot distinguish between reality and virtuallity anymore?
we do not know whether we live in a VR space
 
Augmented Reality opens up the possibility for all of us to become like gods, to have access to all the worlds information at a single glance or gesture, to connect us all up on a deeper and deeper level
I am not talking about living in VR space
 
AR... how do you think about 9/11 in usa?
there are conspiration theories that it might have been AR, no planes at all
since i did not see it with my own eyes, it could be possible for me
the technology is fine enough to do the task even in a live broadcast
 
7:33 AM
I've never heard of such a thing.. how could it be so. Augmented Reality on a mass scale, you cannot permanently destroy the physical with the augmented
You cannot bury dead bodies if it wasn't a physical occurence
Much too many witnesses, too many stories
 
indeed the towers were destroyed... in a very short time
 
We live in a time where as much as govt would like to control us, they are the ones reacting.. the world cannot be controlled, nature's will is ordered chaos
 
i don't know whether there were planes or not, however, i'm pretty sure the planes did not cause the collapse
 
Possibly not.
The military has a rule that it abides by to sacrifice some in favor of the whole
 
yeah the chaos... the towers collapsed in about 12 seconds. according to the height i calculated 9.24 seconds in a vacuum
 
7:37 AM
They certainly used 9/11 to further their own agenda, invade countries, enact laws
 
yes, the latter is the most important
legitimation to start wars is just a nice side-effect
 
This day in age it is very hard to keep a secret
Leaks, anonymity
 
i heared, many us people don't believe the official story and claim a new investigation
it's a really weird story
 
It's rubble now.. what is to investigate
Not that I am against it at all
It will be mostly hearsay
And governments will deny deny deny
 
of course
 
7:42 AM
This is why we need new technologies to flourish
To even moreso give the power back to the people
 
they will even shoot on their own people when they go onto street like in france
 
They are in the business of self-preservation, by whatever means
 
you can support rebells in ukrain and syria, however, in the west contrys they are terrorists when they try to putsch the goverment
 
The only hope is to rise up out of it as a collective
 
unfortunately this does not even work in europe yet
 
7:44 AM
It is ridiculous, stories told to further their own agenda
I have no faith in government to fix our problems
 
i have the hope germans and french people come together, italian, spainish english people as well
 
History has shown what a farce that to be
 
history is repeated periodically
 
This is what I am saying with augmented reality and virtual reality
If we could FEEL what we are doing, to the earth, to each other, we would stop doing it
 
eugenics was nothing new when it started in germany 1933. and it did not end with the ww2
feminism and genderism is a hidden form as well
 
7:48 AM
That is still surface level thinking
 
they want to destruct families
we could feel it, and we often do. however, the economy brings us to sleep
 
We need to realize we are not seperate from one another
An emergence of spirtuality and technology
Connecting us up as one world, one species
If not that, then for what
 
there are raising many esoteric people the last years
 
Religion is fading away, cultural relativity has never before been so apparent
 
i also experience that i have access to some information of the collective consciousness and knowledge
 
7:52 AM
We have to use technology to make the unconscious conscious
 
are you a christ?
 
what do you mean
am i a christian?
 
yes christian
 
Not at all... all religions are flawed, all religions are cultural narratives fit to their particular time and place
There are no answers
 
i beleive the base of all religions is the same and somehow true. but they were misused to rule people
i believe in the creation of life
 
7:56 AM
I agree.. every culture has had some form of religion or worship to a higher deity, to something greater than ourselves
 
the biological organism is much more complicated as the best computer
 
But they are all just stories.. monotheistic religions in the west, polytheistic in the east
Much, yes
For now
 
we can even find remnants of polytheistic religion in the bible
 
What do you think of the singularity, per say?
Moore's law of ever increasing computing speeds
 
which singularity do you mean?
big bang, black hole?
 
7:59 AM
To humanize the unknown is a fallacy I think
But it is our way to relate, our own way, because it is ineffable
 
04:00 - 08:0008:00 - 10:00

last day (15 days later) »