last day (16 days later) » 

5:06 PM
0
Q: Extra ":" at the end of output from sudo su -c ls, only when globbing is used

A. AbramovUsing adb shell to run commands on an android device, I get different results when running ls with or without a wildcard ( globbing, i.e * ). When running ls without a wildcard, the last path is displayed properly. When running ls with a wildcard, the path is displayed with an : in the end of it ...

 
ls doesn't do wildcard handlng at all. bash changes ls *.txt to ls a.txt b.txt c.txt before ls is even started. (This is unlike Windows, where command-line parsing is done by the program being started, even if usually by some standard-C-library bits that the user doesn't write or have control of unless they make an active attempt to do so).
 
@CharlesDuffy I understand, so ls runs separately on each one of these because of globbing; however, I still dont understand why ls on the folder would return a different name than ls on the file within that folder. Do you know why that happens?
 
...beyond that, the question of why ls /tmp/a returns /tmp/a, whereas ls /tmp just returns a... well, that's normal ls behavior; it's what it's always done. Do I need to go find the POSIX spec where it's defined?
 
@CharlesDuffy my issue is specifically with the last file: /data/data/com.kauf.wrapmyFaceFunphotoeditor/files/DV-com.co‌​m.kauf.wrapmyFaceFun‌​photoeditor-2020-05-‌​17-17-44-30-DEBUG.tx‌​t: it has an : in the end of it.
 
Ahh -- that's a helpful clarification. If your remote shell has the bash version of printf, I'd suggest running printf '%q\n' /path/to/*; if it doesn't, maybe printf '\0\n' /path/to/* and then pipe that result into hexdump, open it in a hex editor, &c. to see if the :s are there.
 
5:06 PM
@CharlesDuffy question edited, also I've checked, the filename does not have an actual : in it; it is as shown when I run the ls without the wildcard.
 
(btw, as an aside, ls generally shouldn't be used programmatically at all; if output is for a script's consumption rather than a human's, printf '%s\0' is very much the better tool, as it can unambiguously represent all possible filenames in a form that doesn't change based on exactly which vendor's POSIX-compliant version of printf is in use).
My question is not just whether it contains a :, but what it does contain. ls's behavior in presence of nonprintable characters is undefined by the POSIX standard and varies between implementations, as the above-linked wiki page describes; thus, the value of using a hex editor to view a dump in a printable form.
BTW, consider adb shell sh -xc 'su -c "ls ..."' to get a log of the exact ls invocation after the remote shell has expanded the glob.
...with respect to the `printf '%s\0' formulation, by the way -- it puts a NUL after each filename. That's the best choice for the job because NULs can't exist in UNIX filenames, so it's able to represent all the names that are possible. However, the disadvantage -- and it's a significant one -- is that NULs are (typically, in historical UNIX terminals) nonprinting characters, so they aren't actually visible.
 
I'm very open to replacing ls with something better, the printf doesn't work for me
 
...now, if your _local_ shell is bash, you can do something like the following:

```
while IFS= read -r -d '' filename; do
printf 'Remote file: %q\n' "$filename"
done < <(adb shell sh -xc "printf '%s\0' /path/to/*.whatever")
```
that's using printf '%s\0' on the adb side (which all POSIX-compliant versions of printf are required to support), and printf '%q\n' on the local side (a ksh/bash extension that renders nonprintable content visible).
 
Ah man I wish I was better in bash, I'm getting some bad syntax error: /system/bin/sh: syntax error: unexpected '< '
Thank you for sticking with me
 
so, that means you've got /bin/sh instead of bash
which is pretty normal on Android
 
5:12 PM
oh
 
<(...) is a bashism.
 
lol
 
that's why the above code tries to use a local copy of bash and rely on the remote system having only things sh has
which means you need actual real bash on your dev box, but hopefully that's not such a problem?
 
Sadly the devbox is pretty much out of my control
 
okay, in that case, let's go back to the thing I proposed earlier, saving the content from the remote system to a file and inspecting it with a hex editor
adb shell sh -xc "printf '%s\0' /path/to/*.whatever" >local_file.log should create a local_file.log
if the filenames aren't confidential, could you upload that to a binary-safe pastebin (ix.io, for example)?
or if you have a Python interpreter handy...
 
5:15 PM
Doing it
Python i'm comfortable with
 
...you can do something like: open('local_file.log', 'r').read().split('\0')
and see how Python repr()s the result.
 
Will do
My printf is a little weird
adb shell sh -xc "printf '%s\0' /data/data/com.kauf.wrapmyFaceFunphotoeditor/files/DV-*" > local_file.log

returns:
+ printf
See printf --help
printf: Needs 1 argument

to what I assume is stderr
 
Hmmm, let me think for a minute.
...what's the local shell you're using? (Windows cmd.exe? sh? something else?)
 
windows cmd, yes
 
hmmmm. Unfortunately, I don't know the quoting rules there, so I don't know how to write a command that it's not going to munge.
...and you can't install cygwin, or WSU bash-on-Ubuntu-on-Windows, or such?
 
5:19 PM
I'll try to quote better or install something that makes this easier
on it
 
Hmm. Actually, you have Python
we can write Python code for this :)
 
oh
like, os.system? subprocess?
 
What I'm trying to accomplish is:

subprocess.call(['adb', 'shell', 'sh', '-xc', ''' printf '%s\0' /data/data/com.kauf.wrapmyFaceFunphotoeditor/files/DV-* '''])
...blegh, don't know how to make code-quoting work right in chat here.
 
Trying
I get an embedded null character
 
Ah, right
make it r'''
so that the backslash is literal
 
5:22 PM
this might be related to the ls undefined behavior
oh ok
printf: Needs 1 argument
 
...bizarre. The -c should mean that the argument right after it gets held together
 
That's also what I know, but then again there's at least one thing I know that's incorrect
 
...then again, this is Windows, and as I described in my very first comment, in Windows each command controls how it does command-line parsing
 
maybe i can run it on the machine
after adb shell
sh -xc "printf '%s\0' /data/data/com.kauf.wrapmyFaceFunphotoeditor/files/DV-*" > local_file.log
 
the > should be local instead of remote
so adb shell >local_file.log
then you should be able to just do the printf '%s\0' /data/.../DV-* on another line
and then feed it an EOF
 
5:26 PM
alright, trying
 
...jumping back to the thought I was in the middle of earlier: whereas on UNIX, the shell=False behavior of subprocess code is always 100% correct (because an array of C strings is exactly how things gets passed between processes), on Windows, Python with shell=False basically has to assume that the program being run will use the default Windows libc escaping mechanisms when it encodes its argument list to be something the receiving process will parse.
 
Ahhhh I can't create files on that machine, so running it locally is an issue
 
Huh. No %TMPDIR% or such?
(I'd need to look up what the actual environment variable name is, but there should be a variable that points to a temporary place you can always write things to, even if they're all deleted when you log out)
 
The windows machine isn't an issue, I can write files anywhere
That's what I'm on right now
 
...the goal is to write to a file local to your Windows machine
 
5:32 PM
I can write it to the android machine and then adb pull it
Am I not understanding you?
 
Either that, or adb shell >foo isn't doing the redirection to foo locally. (On UNIX, foo will be opened for write before adb is ever started, so adb has no option to make that a remote redirection... but then, you're on Windows, and I don't really know Windows).
 
Lesson learned UNIX > windows
So the parser is my issue
Weird, even windows libc shouldn't be adding stuff to the result.
Is there any sh command I can use thats not printf or ls to achieve this?
 
Hmm. Let's review what commands you do have on your Android system.
Do you have base64, or uuencode?
(printf '%s\0' shouldn't be a problem in and of itself; what's problematic is trying to get its output over to somewhere we can review it without being munged on the way)
 
which base64 looks like I have it
same for uuencode
 
okay, great. So, if you use just use adb shell to get a remote shell, does printf '%s\0' /path/to/DV-* | base64 give you something you can copy-and-paste?
 
5:37 PM
L2RhdGEvZGF0YS9jb20ua2F1Zi53cmFwbXlGYWNlRnVucGhvdG9lZGl0b3IvZmlsZXMvRFYtKgA=
yes
 
(aside: I've been meaning "local" to mean "local to your dev host", and "remote" to refer to the Android device you're trying to interop with)
 
yessss
 
okay, going to ask Python's base64 module to decode that...
 
There's a space
after the *
 
hmm, it didn't expand the wildcard
I read that as a NUL after the *
and we asked printf to spit out NULs after each file
 
5:38 PM
I run that on the android device
 
what's surprising here is that it didn't get expanded
 
one second ill run that on the windows
 
though if we didn't use su, maybe that's because we don't have the permissions
(to read the location with those filenames)
 
adb shell "printf '%s\0' /data/data/com.kauf.wrapmyFaceFunphotoeditor/files/DV-* | base64" returns exactly the same
>That is on the local windows machine
 
I suspect that's because we need su to have permission to read that location
 
5:39 PM
ok on it
 
so, let's try that again: su sh -xc 'printf "%s\0" /path/to/DV-*'
err, with the | base64 on the end
...after using adb shell
 
adb shell "su -c sh -xc printf '%s\0' /data/data/com.kauf.wrapmyFaceFunphotoeditor/files/DV-* | base64"

returns: Printf:needs 1 argument
 
take out the -c
just su sh ...
err, waitaminute
 
su: invalid option -- x
 
the -c is right if we have other content quoted
but you didn't quote it there
 
5:41 PM
help me here i'm confused
ok so su -c "sh ...
 
well, depending on the details of your copy of su
I actually think the Android one you're using doesn't quite comply with the letter of how a normal one works
which would explain why the other answer you got didn't work
 
really
 
anyhow, things are a little simpler if we run adb shell, and then run a command on its stdin
 
alright
 
at that point, let's try su -c sh -xc printf '%s\0' /data/data/com.kauf.wrapmyFaceFunphotoeditor/files/DV-* | base64 inside the already-running adb shell
 
5:43 PM
KyBwcmludGYNClNlZSBwcmludGYgLS1oZWxwDQpwcmludGY6IE5lZWRzIDEgYXJndW1lbnQNCg==
 
Hmm. That's too short, smells like the glob failed to expand again
 
thats for
su -c sh -xc printf '%s\0' /data/data/com.kauf.wrapmyFaceFunphotoeditor/files/DV-* | base64

inside the adb shell
The result is + printf
See printf --help
printf: Needs 1 argument
 
oh
right.
the glob is expanding before su runs
 
ohhhh so thats many params for the printf?
 
so, this is where I was saying we need more quotes
su -c sh -xc "printf '%s\0' /data/data/com.kauf.wrapmyFaceFunphotoeditor/files/DV-*" | base64
so, we're quoting things after the sh's -c, not things after the su's -c
 
5:45 PM
I see, because it groups up the parameters
I get the same result
 
...now, that's surprising.
that said, can you run su inside the Android device and get back an interactive shell as root?
that'll let us continue to unbundle
 
yesss
yes i can
Ok I ran su now im root
 
okay, great. Then once you've got that root shell, printf '%s\0' /path/to/DV-* | base64 in there
 
so, sh -xc "printf '%s\0' /data/data/com.kauf.wrapmyFaceFunphotoeditor/files/DV-*" | base64
 
don't need the sh anymore
the whole point of the sh was to have su start a shell, so that inner shell could expand the glob
but we already have an inner shell with its privileges escalated now
 
5:47 PM
Ohh so that simplifies things
 
yup. And if you want to emulate the effect of the -x argument to sh, you can run set -x in that inner shell manually
that'll tell it to log commands as it's about to run them to stderr
 
Inside the adb interactive shell, after running su ( and whoami -> root ), I ran:
printf '%s\0' /data/data/com.kauf.wrapmyFaceFunphotoeditor/files/DV-* | base64

The result was the base64 of:
/data/data/com.kauf.wrapmyFaceFunphotoeditor/files/DV-*
 
so you can see how they get expanded wrt. globbing &c.
interesting.
does ls /data/data/com.kauf.wrapmyFaceFunphotoeditor/files/DV-* expand properly?
 
Oh it actually returns a completely different result
ls: /data/data/com.kauf.wrapmyFaceFunphotoeditor/files/DV-*: No such file or directory
Thats so weird
 
that's consistent with what we're getting from printf
(because globs don't expand when they don't have any results)
 
5:50 PM
wait wait i think the path is wrong
ls /data/data/com.kauf.wrapmyfacefunphotoeditor/files/DV-*
that works
returns with :
 
(a-yup, case-sensitive filesystems are a thing)
 
lol my bad
 
okay, and if you switch back to the printf, with that same correction to the name?
 
L2RhdGEvZGF0YS9jb20ua2F1Zi53cmFwbXlmYWNlZnVucGhvdG9lZGl0b3IvZmlsZXMvRFYtY29t
LmthdWYud3JhcG15ZmFjZWZ1bnBob3RvZWRpdG9yLTIwMjAtMDUtMTctMTctNDQtMzAtREVCVUcu
dHh0AC9kYXRhL2RhdGEvY29tLmthdWYud3JhcG15ZmFjZWZ1bnBob3RvZWRpdG9yL2ZpbGVzL0RW
LWNvbS5rYXVmLndyYXBteWZhY2VmdW5waG90b2VkaXRvci0yMDIwLTA1LTE3LTE3LTQ1LTA5LURF
QlVHLnR4dAAvZGF0YS9kYXRhL2NvbS5rYXVmLndyYXBteWZhY2VmdW5waG90b2VkaXRvci9maWxl
cy9EVi1jb20ua2F1Zi53cmFwbXlmYWNlZnVucGhvdG9lZGl0b3ItMjAyMC0wNS0xNy0xNy00NS0w
OS1JTkZPLnR4dAAvZGF0YS9kYXRhL2NvbS5rYXVmLndyYXBteWZhY2VmdW5waG90b2VkaXRvci9m
Much longer
 
...copies/pastes it into a Python interpreter to decode...
 
5:52 PM
/data/data/com.kauf.wrapmyfacefunphotoeditor/files/DV-com.kauf.wrapmyfacefunphotoeditor-2020-05-17-17-44-30-DEBUG.txt
 
yup, no :, looks perfectly reasonable to me.
 
no : in the decoded base64
Yes yes
Ok now I need to unpack this
This works ok without base64
 
indeed. So, I'd like to try something: What happens if you put a ;echo after the ls? Does the : still show up? If so, is it before or after the extra newline on the end?
 
ls /data/data/com.kauf.wrapmyfacefunphotoeditor/files/DV-*; echo

Still has `:` in the end, and then an extra newline
its before
 
okay, that answers that; it's coming from ls, not from the shell...
...does your Android system have strace installed?
 
5:54 PM
nope
 
shoot. Would have been handy to get a syscall-level trace of what its copy of ls was up to.
 
Heres something thats interesting
when I run
adb shell su -c printf '%s\0' /data/data/com.kauf.wrapmyfacefunphotoeditor/files/DV-*

I get a 0 in the end
i actually get a lot of 0
each file ends with a .txt0
 
that's just the \0 being munged before it gets to the shell
backslash handling when things go through extra escaping layers is generally pretty weird.
part of why f/e I usually encourage people writing regexes to write [.] instead of \., because it's too easy to have something you don't expect just delete the backslash.
and if it gets changed from '%s\0' to '%s0', then you get extra 0s everywhere.
 
That makes sense. You've been very helpful and I'm learning a lot here
How would you go about understanding why does ls does what it does?
 
Hmm. I'd probably try to either get a copy of strace installed, or look at the source of the version of busybox that that copy of ls comes from (assuming it's busybox, but on Android that's more likely than not true). Ideally, both.
...as the strace logs generally will make it easier to know where to look in the source -- if there's a syscall that fails before the : is written, or just surrounding context that gives hints on what it's doing at that point in time, all of that can be useful.
 
6:02 PM
I see. well, printf '%s\\n' produces percisely the results I wanted, so for now I'd like you to post that as an answer so I can award you your well deserved bounty
 
Added; hopefully that works.
 

last day (16 days later) »