last day (15 days later) » 

10:35 PM
0
A: Pass a list of filenames from a .txt to a find and copy command

jaba10You can see where your files are with this: for FILE in $(find /Images/JPG -type f | grep -f <(sed -r "s/(.*)\.xml/\1/" /Sites/csv/list.xml)); do mv /Sites/csv/$(basename $FILE).xml $(dirname $FILE) done Explanation: find /Images/JPG -type f List all the files since the directory /Image...

 
I understand that grep is going line-by-line through my file list and piping it to find the corresponding image. Does this command also move the xml file? How does that work? Sorry, I'm a bit new to this still.
taking a look at this now! Thank you for breaking it down for me. This looks positive. I'll keep you posted!
 
I replaced the xargs for a for loop. I'm not a very good friend of xargs.
 
My only problem here is that basename produces the full file path and dirname produces nothing, leading to the wrong path. Not sure why $(basename FILE) isn't outputting as you suggest
error - grep requires an argument --f?
 
When I try it works. It must be because i wrote /Sites/csv/list.xml instead of /Sites/csv/file_list.xml .
 
I get the following error now: move.sh: line 8: syntax error near unexpected token (' move.sh: line 8: find Finals/New_Test/images -type f | grep -f <(sed -r "s/(.*)\.xml/\1/" /Finals/New_Test/xml.txt) | '
 
10:35 PM
I opened a chat, There's a pipe that isn't in my code. Shoe mw your script and it'll be my pleasure to help you
 
not sure how to access the chat? Also, I'm running this within a .sh file - does that matter?
 
It doesn't. Could you show me line 8 in your script?
 
excellent! Thank you again so much. I'll copy the code:
for FILE in $(find Finals/New_Test/images/ -type f | grep -f <(sed -r "s/(.*)\.xml/\1/" Finals/New_Test/xml.txt)); do
mv Finals/New_Test/$(basename $FILE).xml $(dirname $FILE)
done
the move.sh script is in the Sites/csv folder... the full directory is - Sites/csv/Finals/New_Test and images are in Sites/csv/Finals/New_Test/images
 
Let's see in a console what shows any command:
first: find Finals/New_Test/images/ -type f
 
Finals/New_Test/images//.DS_Store
Finals/New_Test/images//day1/DF_00001.jpg
Finals/New_Test/images//day1/DF_00002.jpg
Finals/New_Test/images//day1/DF_00003.jpg
Finals/New_Test/images//day1/DF_00004.jpg
Finals/New_Test/images//day2/DF_000011.jpg
Finals/New_Test/images//day2/DF_000012.jpg
Finals/New_Test/images//day2/DF_000013.jpg
Finals/New_Test/images//day2/DF_000014.jpg
 
10:38 PM
it must show all the files (.jpg) since Finals/New_Test/images/
 
yes... I removed the trailing / in images/ and it shows all of the test files
8 images across 2 sub folders
 
well, first of all, due to there are files that aren't .jpg, you can change the find to this:
find Finals/New_Test/images/ -type f -name "*.jpg"
Then, you can try:
sed -r "s/(.*)\.xml/\1/" Finals/New_Test/xml.txt
 
sed: illegal option -- r
usage: sed script [-Ealn] [-i extension] [file ...]
sed [-Ealn] [-i extension] [-e script] ... [-f script_file] ... [file ...]
 
It must return all the names in your txt without the .xml
that's weird
What s.o. do you use?
 
I was able to do that succesfully earlier with:
sed 's/.xml//' Finals/$1/xml.txt > Finals/$1/image_name.txt
 
10:43 PM
great, that works too
 
MacOsX Terminal? nano to edit in a .sh file
so, now I have a file with just the image names
 
so, you can change it to that way
well, maybe that was because I use arch Linux with some another version of sed
 
so, grep -f Finals/New_Test/image_name.txt
returns nothing - I have to cancel
 
find Finals/New_Test/images/ -type f -name "*.jpg" | grep -f <(sed 's/.xml/\1/' Finals/New_Test/xml.txt
it returns nothing because you need an input
it is the one returned by find
I think I missed a )
 
copied and still returns nothing
ah
sed: 1: "s/.xml/\1/": \1 not defined in the RE
thank you again for all of this - I am learning alot
(looking things up as you're typing)
 
10:47 PM
sorry sorry
put your sed, I just mixed my command with yours. :S
 
not sure what you mean - where does the :S go?
 
ind Finals/New_Test/images/ -type f -name "*.jpg" | grep -f <(sed 's/.xml//' Finals/$1/xml.txt)
find instead of ind
:S is just to express something, it doesn't mean anything
Does it return what you want?
 
very nice. Yes it does
 
Great!
 
putting that back in the script
 
10:50 PM
Now, you could modify your script with those changes and it should work
Nice
 
close with an additional ) correcT?
hmm
for FILE in $(find Finals/New_Test/images -type f -name "*.jpg" | grep -f <(sed 's/.xml//' Finals/$1/xml.txt)); do
mv Finals/New_Test/$(basename $FILE).xml $(dirname $FILE)
done
this gave another syntax error
move.sh: command substitution: line 3: syntax error near unexpected token `('
move.sh: command substitution: line 3: `find Finals/New_Test/images -type f -name "*.jpg" | grep -f <(sed 's/.xml//' Finals/$1/xml.txt)'
 
I didn't understand this: __
close with an additional ) correcT?__
 
the full script opens with a for FILE in $(..... but I don't see where that closes. So I added a closing before the );do
 
Do you use bash?
 
yes
executing with sh
 
10:54 PM
for FILE in $(find /Images/JPG -type f -name "*.jpg" | grep -f <(sed 's/.xml//' /Sites/csv/file_list.xml)); do
mv /Sites/csv/$(basename $FILE).xml $(dirname $FILE)
done
that's an example, the filenames could be incorrect, but that's the sintax
 
yes... so I added the second ) to the find and grep commands that worked in isolation
I'll substitute once more and see if I can get it to work
 
please
 
hurumph... back to the original error:
move.sh: command substitution: line 3: syntax error near unexpected token `('
move.sh: command substitution: line 3: `find Finals/New_Test/images -type f -name "*.jpg" | grep -f <(sed 's/.xml//' Finals/New_Test/xml.txt)'
here's my code with substitutions:
for FILE in $(find Finals/New_Test/images -type f -name "*.jpg" | grep -f <(sed 's/.xml//' Finals/New_Test/xml.txt)); do
mv Finals/New_Test/$(basename $FILE).xml $(dirname $FILE)
done
seems identical to me...
 
I can't see which '(' refers to
If it works for you:
find Finals/New_Test/images -type f -name "*.jpg" | grep -f <(sed 's/.xml//' Finals/New_Test/xml.txt)
 
yes. works perfect
 
11:00 PM
you can save that string in a variable
example:
MYVAR=$(find Finals/New_Test/images -type f -name "*.jpg" | grep -f <(sed 's/.xml//' Finals/New_Test/xml.txt))
for FILE in $MYVAR; do
....
so, its easier to read
Would be the path where you're running the script?
 
still getting an error this way:
 
Think that you're using relative paths
 
yes. I think I am too
my move.sh file lives:
~/Sites/csv/
when I test in the console, I've cd'd to /csv
and when I run find and grep, I'm starting at Finals
which is really ~/Sites/csv/Finals
and the full paths are:
~/Sites/csv/Finals/New_Test/xml.txt
and ~/Sites/csv/Finals/New_Test/images/day1/DF0001.jpg
errors:
move.sh: command substitution: line 12: syntax error near unexpected token `('
move.sh: command substitution: line 12: `find Finals/New_Test/images -type f -name "*.jpg" | grep -f <(sed 's/.xml//' Finals/New_Test/xml.txt)'
sorry this is taking so long :(
 
That's not a problem for me. I like do this
when you say:
when I test in the console, I've cd'd to /csv
 
my pwd is ~/Sites/csv
 
11:07 PM
That's different of ~/Sites/csv
~ is your home
Does exist both ~/Sites/csv and /csv?
 
I'm running console from ~/Sites/csv and I'm running move.sh from ~/Sites/csv
same
 
great, that's not the problem so
You can try all the code in your console instead inside the script
 
okay. will do that now
 
Or you can try replacing: <(sed 's/.xml//' Finals/New_Test/xml.txt) in this way
sed 's/.xml//' Finals/New_Test/xml.txt) > /tmp/myfile.txt
for FILE in $(find /Images/JPG -type f -name "*.jpg" | grep -f /tmp/myfile.txt); do
mv /Sites/csv/$(basename $FILE).xml $(dirname $FILE)
done
Is there something more before that code in your script? Maybe something previous
If it's not so long, you could show me all your script
 
sure - I was trying to dynamically pass in the name of "New_Test" because I have many folders to run this on
but, to get it to work, I've been hard-coding to get rid of variables...
here's the entire script
#!/bin/bash

function write (){

ls ~/Sites/csv/Finals/$1/xml > ~/Sites/csv/Finals/$1/xml.txt
sed 's/.xml//' Finals/$1/xml.txt > Finals/$1/image_name.txt

for FILE in $(find Finals/New_Test/images -type f -name "*.jpg" | grep -f <(sed 's/.xml//' Finals/New_Test/xml.txt)); do
mv Finals/New_Test/$(basename $FILE).xml $(dirname $FILE)
done

}


read -p "XML Source Folder Name: " folder
write $folder
echo finished
New_Test would typically come from $1
but I commented out the top two lines for our troubleshooting and changed the $1 to a static "New_Test"
and my response (uncommented)::
XML Source Folder Name: New_Test
move.sh: command substitution: line 3: syntax error near unexpected token `('
move.sh: command substitution: line 3: `find Finals/New_Test/images -type f -name "*.jpg" | grep -f <(sed 's/.xml//' Finals/New_Test/xml.txt)'
finished
that's the whole thing :)
 
11:19 PM
I can't figure out yet where the problem is
 
shoot :( I feel like it's so close. I'm looking up command substitution and seems there could be something with passing arguments across the pipe and additional syntax?
ah. I may have solved it
I put a `` around the grep -f <(...)
so... grep -f <(.....));
 
great
 
no syntax error - but the code still didn't work
this is encouraging though:
move.sh: line 3: DF_00001.jpg
DF_00002.jpg
DF_00003.jpg
DF_00004.jpg: No such file or directory
finished
can I echo out the destination and source for the mv command?
 
line 3 is...?
 
for FILE ...
I think the mv source and directory may still be wrong
 
11:27 PM
sed 's/.xml//' Finals/New_Test/xml.txt > /tmp/myfile.txt
FILE_NAMES="$(find Finals/New_Test/images -type f -name "*.jpg" | grep -f /tmp/myfile.txt)"
for FILE in "$FILE_NAMES"; do
mv Finals/New_Test/$(basename $FILE).xml $(dirname $FILE)
done
You can write it in that way, I think could be easier to know what's wrong
 
does mv -v print out the source and destination?
 
would be that your chell doesn't recognize the way: $() , instead of this, you yan try `` , is just the same
 
okay - I'll copy your code above and try... one moment. THANKS
 
yes, it does
 
I tried to add mv -v and it didn't output anything:(
received a strange error:
usage: dirname path
usage: mv [-f | -i | -n] [-v] source target
mv [-f | -i | -n] [-v] source ... directory
 
11:31 PM
let's debug it with echo
 
great
I'll go back to our first syntax - as I think that was REALLY close
can I echo the move ?
actually - I'll do whatver you advise :)
 
sed 's/.xml//' Finals/New_Test/xml.txt > /tmp/myfile.txt
echo "/tmp/myfile.txt"
cat /tmp/myfile.txt

FILE_NAMES="$(find Finals/New_Test/images -type f -name "*.jpg" | grep -f /tmp/myfile.txt)"
echo $FILE_NAMES

for FILE in "$FILE_NAMES"; do
echo Finals/New_Test/$(basename $FILE).xml
echo $(dirname $FILE)
done
that code doesn't try to move anything, just echo what would move
 
/tmp/myfile.txt
DF_00001.jpg
DF_00002.jpg
DF_00003.jpg
DF_00004.jpg
Finals/New_Test/images/day1/DF_00001.jpg Finals/New_Test/images/day1/DF_00002.jpg Finals/New_Test/images/day1/DF_00003.jpg Finals/New_Test/images/day1/DF_00004.jpg
Finals/New_Test/DF_00001.jpg DF_00002.jpg DF_00003.jpg DF_00004.jpg.xml
usage: dirname path

finished
so... those are the correct image paths
I don't see the paths to the xml docs though
 
me neather
 
myfile.txt just has the image names in it
so, file names and paths look good
 
11:38 PM
yes
 
I updated:
for FILE in "$FILE_NAMES"; do
echo Finals/New_Test/xml/$(basename $FILE).xml
echo $(dir
echo $(dirname $FILE)
done
and now I get:
 
I see, the loop for isn't striping FILE_NAMES
let's try with while
 
/tmp/myfile.txt
DF_00001.jpg
DF_00002.jpg
DF_00003.jpg
DF_00004.jpg
Finals/New_Test/images/day1/DF_00001.jpg Finals/New_Test/images/day1/DF_00002.jpg Finals/New_Test/images/day1/DF_00003.jpg Finals/New_Test/images/day1/DF_00004.jpg
Finals/New_Test/xml/DF_00001.jpg DF_00002.jpg DF_00003.jpg DF_00004.jpg.xml
usage: dirname path
so.. I get the path for the first xml, but the script isn't appending .xml to the jpg
all other entries have dropped the path
 
yes, it takes all the string as one thing
took out the "" in "$FILE_NAMES" please
sed 's/.xml//' Finals/New_Test/xml.txt > /tmp/myfile.txt
echo "/tmp/myfile.txt"
cat /tmp/myfile.txt

FILE_NAMES="$(find Finals/New_Test/images -type f -name "*.jpg" | grep -f /tmp/myfile.txt)"
echo $FILE_NAMES

echo ""

for FILE in $FILE_NAMES; do
echo Finals/New_Test/$(basename $FILE).xml
echo $(dirname $FILE)
done
 
/tmp/myfile.txt
DF_00001.jpg
DF_00002.jpg
DF_00003.jpg
DF_00004.jpg
Finals/New_Test/images/day1/DF_00001.jpg Finals/New_Test/images/day1/DF_00002.jpg Finals/New_Test/images/day1/DF_00003.jpg Finals/New_Test/images/day1/DF_00004.jpg
Finals/New_Test/xml/DF_00001.jpg.xml
Finals/New_Test/images/day1
Finals/New_Test/xml/DF_00002.jpg.xml
Finals/New_Test/images/day1
Finals/New_Test/xml/DF_00003.jpg.xml
Finals/New_Test/images/day1
Finals/New_Test/xml/DF_00004.jpg.xml
Finals/New_Test/images/day1
finished
okay. this is looking promising!
 
11:44 PM
yeah!
add the mv and it would work
 
and it did!
 
I'm so happy!!
 
I can't believe you stuck with me through all of that!
 
it's my pleasure, I'll edit the answer in the way that it works
 
wow. THANK YOU THANK YOU THANK YOU
 
11:49 PM
you're very welcome
 
not sure how I could repay the favor - but please let me know if there's ever anything I can do. I'm strong in SQL, T-SQL, PHP and javascript
I'll make sure to credit you back on SO
 
I'll ask you if needed, that's for sure
we can delete our comment to let the post clearer
 

last day (15 days later) »