last day (14 days later) » 

1:09 AM
2
A: How to read a binary file which has some text string in it in using shell script?

Corentin PeuvrelYou should try something with strings, it only keep printable ASCII characters from your file : strings - 142490.1 | awk -F '.' -v timestamp="$(date +%s)" '/^@client/ && $2 < (timestamp - 60)*1000 {print}' This awk script may be too specific to this example : it look at the field between the...

 
@CorentinPeuvrel I tried this but it printed out bunch of stuff as well which doesn't start with @client. Any idea what could be wrong?
 
Try to replace the awk script by /^@client/ && $2 < (timestamp - 60)*1000 {print}. I didn't checked that it was a line starting with @client
 
@CorentinPeuvrel Can you edit that as well for the sake of completeness?
@ThomasDickey I see undestood now.
@CorentinPeuvrel After I tried your recent suggestion, it's not printing out anything. Looks like I might be off what I was trying to do. I guess I need to print out @client strings whose difference between current timestamp and timestamp in the string is greater than 1 minute.
 
Try strings - 142490.1 | awk -F '.' '/^@client/ {print $2}', just to see if, without the check on the timestamp, we really get something (and, by adding the $2 after the print, we'll see what is identified as the timestamp)
 
I just tried it and I didn't got anything back. looks like something is wrong. When I opened the file, I can see there are strings with @client.
 
1:09 AM
Maybe the line doesn't really start with '@client'. Could you try strings - 142490.1 | awk -F '.' '/@client/ {print $2}' (the same except we just match lines with "@client" instead of starting with it)
 
@CorentinPeuvrel Yup now it worked fine. Indeed it doesn't starts with @client.
 
Ok, so it's a little more complicated, we cannot assume that the timestamp will be the second field, so we have to do something like what @ThomasDickey proposed. I cannot comment yet, but in his script, I would have used a awk variable for $TS with awk -v sysTs="$(date +%s)000" '/@client/ { ts = gensub("^.*@client\.","", "", $0); sub("\..*$","",ts); if ( ts >= sysTs - 60 and ts < sysTs ) { print $0; } }' (I also merge the first two instructions in one gensub)
Btw, I think the timestamp condition is the opposite of what you said. You wanted line "older" than T-1min, is that it ? So the if should be if ( ts < sysTs - 60)
 
@CorentinPeuvrel let's chat here
 
So, does it works with this ?
 
I got confuse what timestamp comparison I need to do
so now I understood what I need to do
Let's say
if I am running on file name -
strings - 1424911080.1
which is 1424911080
It will always have 10 digits
I need to use this timestamp - timestamp in the string which is in milliseconds as it has 13 digits
and if the difference is greater than 500 ms then print out those strings
 
1:15 AM
So, you're saying that you need to get the timestamp from the file name ?
So you don't care about the timestamp, you just need to know the filename. Is that it ?
 
noo noo
let me explain
If I am running the command on this file name `1424911080.1`, and we have below string in the file -

@client.1424912520832.1522924567881.DC1.b7b6ad-4891-4f-9e7b-04019368

then I need to use this formula -

(1424911080 - 1424912520832) and if the difference is greater than 500 ms, then print out this string.
 
Ok, I think I get it. So you don't have to use date +%s, do you ?
 
no I don't need to
Sorry I was confuse earlier so miscommunicated it
timestamp in the file name is 10 digit
but the timestamp in the string is 13 digit
 
No problem. Maybe this would do the trick : FILE=1424911080.1 ; strings - $FILE | awk -v fileTs="${FILE%.*}000" '/@client/ { ts = gensub("^.*@client\.","", "", $0); sub("\..*$","",ts); if ( ts - fileTs > 500 || ts - fileTs < -500 ) { print $0; } }'
The ${FILE%.*} remove what is after the dot in the filename
 
sure
 
1:28 AM
Did this work ?
 
I think we still need to deal with timestamp conversion right?
Bcoz fileName timestamp is 10 digit which is in second?
and timestamp in the string is of 13 digits which is in milliseconds
so we need to multiply fileTs * 1000
right?
bcoz filetimestamp will always be 10 digit
 
In fact, i did a little "trick" here as fileTs = "${FILE%.*}000", so, in your example 1424911080000
 
aahhh
pretty cool
I was not aware of this
let me try now
I got error as
awk: line 2: function gensub never defined
I ran the exact command on my linux console
 
maybe it's an old version of awk, try : FILE=1424911080.1 ; strings - $FILE | awk -v fileTs="${FILE%.*}000" '/@client/ { ts = $0 ; sub("^.*@client\.","", ts); sub("\..*$","",ts); if ( ts - fileTs > 500 || ts - fileTs < -500 ) { print $0; } }'
 
Yes it did worked and printed out bunch of stuff
now let's do whether they are actually 500 ms old
 
1:35 AM
It's a good beginning !
 
yeah
F@client.1424912580790.48220932383.DC1.8d2ad5aa-4e12-4d6f-88f6-59da6df885fb
G@client.1424912580804.6556853530264.DC1.0f7bc5b1-dd36-432b-b0b0-6a18ed59c760
H@client.1424912580805.1022325519919.DC1.4c8f2630-07bc-451d-a7a5-d4c00a6b569b
G@client.1424912580814.219243133.DC1.65747bdf-0510-4c18-a2a7-bcc4da9c2a4f
G@client.1424912580810.585034345805.DC1.123cc92a-055e-4dd5-a328-8b975529ae31
sample string which got printed out
 
It's on the file 1424911080.1 ?
 
1424912580
on this
one
FILE=1424912580.4 ; strings - $FILE | awk -v fileTs="${FILE%.*}000" '/@client/ { ts = $0 ; sub("^.*@client\.","", ts); sub("\..*$","",ts); if ( ts - fileTs > 500 || ts - fileTs < -500 ) { print $0; } }'
this is what I ran
 
Yeah, apparently, it's indeed what we want :)
I should update my message with this
 
sure
one more thing
is there any way to print the total count of these at the end, so that i don't need to count whole stuff one by oe?
meaning how many of the string it printed out?
 
1:40 AM
You mean count the number of lines ?
add this at the end : | wc -l
 
I mean to say, print out the total strings it has scanned and how many is actually 500 ms old
something like that
is this possible anyhow?
 
To have sth like a percentage of >500ms old ?
You have to do it in 2 steps I think :
 
yeah like there were 10000 entries present with @client.ts
and we found 500 entries with 500 ms old
 
FILE="1424911080.1" ; tot=$(strings - "$FILE" | grep '@client' |wc -l) ; old=$(... | wc -l ) ; echo "old : $(( old * 100 / tot ))%"
with "..." the actual code above
 
so something like this
FILE="1424912580.4" ; tot=$(strings - "$FILE" | grep '@client' |wc -l) ; old=$(FILE=1424912580.4 ; strings - $FILE | awk -v fileTs="${FILE%.*}000" '/@client/ { ts = $0 ; sub("^.*@client\.","", ts); sub("\..*$","",ts); if ( ts - fileTs > 500 || ts - fileTs < -500 ) { print $0; } }' | wc -l) ; echo "old : $(( old * 100 / tot ))%"
you mean to say
 
1:48 AM
Yep
at one detail
old=$(string - $FILE | ..... there isn't the FILE=1424912580.4 in it
 
so basically this
FILE="1424912580.4" ; tot=$(strings - "$FILE" | grep '@client' |wc -l) ; old=$(strings - $FILE | awk -v fileTs="${FILE%.*}000" '/@client/ { ts = $0 ; sub("^.*@client\.","", ts); sub("\..*$","",ts); if ( ts - fileTs > 500 || ts - fileTs < -500 ) { print $0; } }' | wc -l) ; echo "old : $(( old * 100 / tot ))%"
 
I posted it if you want
 
yeah looks similar
 
So, everything works as you want ?
 
yeah everything works but I got percentage as 98%
lol
that seems very high what we are doing
does percentage calculation looks right?
just wanted to make sure
one more thing, I am seeing some entries like this -
=@client.SYNC.7795633932.DC1.406a5c1b-e07d-4374-b294-5727a4c3e000
=@client.SYNC.7795633932.DC1.85248348-d8e5-458b-a1f8-815da978bc35
=@client.SYNC.7795633932.DC1.68744dee-d5f3-43d1-bf52-47eb7ddc5992
in which second field is not the timestamp
so is there any way to ignore these?
sorry for asking too much stuff here
once it started working
 
1:54 AM
So theses don't have a timestamp at all ?
 
I realized we have other stuff as well in that file since I cannot scan everything by naked eye
yeah they will not have timestamp
they will have this pattern always
@client.SYNC.
so we need to ignore these
 
I think changing "/@client/" by "/@client.[0-9]/" should do it
And you can 'echo $tot' and 'echo $old' to check the calculation (it's 3AM here, starting to doubt my maths ^^)
 
lol, I think you should go to sleep
I can check with you whenever you are free tomorrow
do I need to add [0-9] after each @client we have in that command?
 
Ok, I'll try to think to log on tomorrow.
 
sure
we can use the same chatroom
 
1:59 AM
No, just old=$(strings - $FILE | awk -v fileTs="${FILE%.*}000" '/@client.[0-9]/
Yep
 
after this change I got 93%
let me play with this
we can chat more tomorrow
in the same chat room
you should go and take some sleep
 
Ok, good night, bye
 
gdnyt
 
 
12 hours later…
2:05 PM
Hi
 

last day (14 days later) »