« first day (1806 days earlier)      last day (1052 days later) » 

mr5
5:22 AM
set(CMAKE_CXX_FLAGS "-D_WEBSOCKETPP_CPP11_STL_ -lboost_system -lcurl -lcrypto -lboost_chrono-mt -lboost_thread-mt -lssl -lcpprest")
This one is causing linker warnings
'linker' input unused
any ideas what I'm doing wrong?
 
6:00 AM
linux?
it seems no need to add boost libraries manually
AFAIK
Why may happen if boost::asio::ip::tcp::socket::cancel() is called while the socket is just sending messages to the remote end?
As per the official document, which says that

basic_stream_socket::cancel causes all outstanding asynchronous connect, send and receive operations to finish immediately.

But it does not say where the data which has not been sent out goes? Must I resend the entire message again? Or just resend the data which has not been sent out (in this case, how could I know how many bytes have sent and how many have not?)?

And I know that this API will only cancel asynchronous operations that were initiated in the current thread.
0
Q: Why may happen if boost::asio::ip::tcp::socket::cancel() is called while the socket is just sending message to the remote end?

JohnAs per the official document, which says that basic_stream_socket::cancel causes all outstanding asynchronous connect, send and receive operations to finish immediately. But it does not say where the data which has not been sent out goes? Must I resend the entire message again? Or just resend t...

 
 
4 hours later…
9:52 AM
@mr5 you link the static versions of boost chrono and thread, but the dynamic version of boost system?
I think that warning normally tells you which linker input was not used
 
 
2 hours later…
11:39 AM
@Mgetz Mostly are binary except the delimiters for each packets.
 
@John how are you treating the data... as text or as binary?
so you're concerned about byte order?
 
@Mgetz The delimiters are textual, e.g: ###BGN***. I need to restore each packet from the received TCP stream.
You see there is no API for std::vector<char> to find the location for specific text(the delimiter for each packet e.g: ###BGN***).
 
let's step back, A) how much data are you storing into the container on average. B) what's the message you want to send to a following programmer? C) can you be sure your delimiter won't show up in the binary data (hint... you can't unless the binary is base64)
 
The data received is something like this: ###BGN### .......BINARY_DATA......... @@@END@@@###BGN### .......BINARY_DATA......... @@@END@@@###BGN### .......BINARY_DATA......... @@@END@@@
 
and this is why you don't roll your own protocol...
 
11:46 AM
@Mgetz A: mostly are less than 64KB, few messages are about 100MB
 
so std::string won't provide any performance advantages then
but seriously if you have an opportunity to change how you're doing this... please do
 
@Mgetz B: to a following programmer? I could catch you. If I understand you correctly, the data is used by many processes, maybe device status, map data, and etc.
 
basically how do you want the programmer that comes in after you to read this
remember the documentation they'll get is probably wrong or misleading
honestly... everytime I see someone roll their own protocol I want to bonk them and tell them to go use protobuf
 
@Mgetz C The probability is very low since the packet head is already very long and there is a value(which is just behind the packet head) to indicate the real length of the packet.
 
well that's good... but I still think you'd be better off using protobuf
 
11:53 AM
@Mgetz It's too slow when encountering large data, e.g: 100MB.
@Mgetz "basically how do you want the programmer that comes in after you to read this remember the documentation they'll get is probably wrong or misleading", I think you mean std::string may mislead the other programers for the data is almost binary data indeed. Am I right?
 
@John Yes... you may want to use std::byte too
@John then you're probably using it wrong... as it's the line protocol for literally all of google... and they are moving around petabytes
 
@Mgetz std::byte is a good idea. But that's for C++17, our compiler does not support it well.
 
that's unfortunate
also you don't need to ping reply to every single message...
 
@Mgetz I used protobuf when using FASTRTPS. But it really seems slow when facing the data with the capacity of 100MB.
@Mgetz "you don't need to ping reply to every single message..." Could you please explain that more in detail?
 
@John this is a ping reply
this is a reply
 
12:07 PM
@Mgetz "ping reply"? "reply"? Sorry, I really could not catch up. Is there any document or article which could be recommended to me to dig it deeper?
 
EVERY SINGLE TIME you click the reply to message button... it pings me
LOUDLY
this is highly unnecessary
 
nwp
There is a mute button in the top right next to "all rooms".
 
I'd rather not... there are legitimate reasons I'd need to be pingged
 
Sorry, I just intend to make my response more clear, because my response may be a little slow.
 
if you're replying to something like 10 messages back.. it makes sense
or if there are multiple conversations going on
but when it's just two people.... it really doesn't
 
12:12 PM
I see. Sorry.
 
np
 
One more question, as we just discussed, std::vector<char> seems to be a better choice. Is there API which could efficiently search specific substring for std::vector<char>, just like std::string::find. And std::find() could not achieve this goal.
 
nwp
Not in the standard library. There is nothing that deals with streams well enough to do that.
 
I mean you could do it as strings until you pull out the binary blob... the issue is that's a waste of a memory copy
and it's very plausible the binary could have the delimiter in it... binary data is basically opaque
 
nwp
std::string_view::find is a building block you can use, but since you don't have the package you will need to manually keep track of where you already searched and back up a bit before the next search in case the delimiter got split up.
 
12:21 PM
I see. Thank you for your generous help.
 
nwp
The better approach would be to ignore the @@@END@@@###BGN### stuff and use the length in the package header which you said exists. That makes everything much easier.
 
Such stuff(@@@END@@@...) is to make the program more robust. If a single packet goes wrong, the other packets are not influenced. I mean I could still restore other packets if a single packet is wrong(maybe, because the program bug).
 
nwp
You are defending against something that will never happen.
Or to rephrase it in something that will probably be more convincing: You're wasting performance on unnecessary stuff.
 
eh... depends on the network
but regardless you need to have a cynical parser
 
nwp
4 bytes size, then your binary is all you need.
 
12:28 PM
that rejects anything outside of it's parameters
@nwp that's dangerous... what if 0xFFFFFFFF comes in?
 
nwp
Then you reserve 4GB of space.
 
@Mgetz Yes, so there is also a CRC32 for each packets.
 
nwp
There are situations where you do need a preamble so you can find the start of a package in a data stream that you caught in the middle, but you don't have that situation. I also remember you are in a local network, not even going via internet.
@John You really shouldn't edit pinging messages after you were told pings are annoying. It pings again on every edit.
 
@nwp so you basically let an attacker DOS you?
 
nwp
Local network. Also ... you don't even need to reserve anything.
They get to send you 4GB of data. There is nothing you can do about that no matter what protocol you use.
 
12:31 PM
Most packets are in a local network, but not all. A few packets are still going via internet.
 
nwp
And receiving 4GB of data should not deny your service.
It's not even an attack, it's a supported feature.
 
it might if you only have 2GB of RAM :P
 
4GB! What a huge memory! The devices are mostly just not more 8GB.
 
nwp
You can limit it to 128MB if that is enough for every message.
 
It's a waste to do so.
 
nwp
12:33 PM
What do you mean waste? You don't waste anything.
 
4GB memory is a waste.
 
if the largest expected message is 100MB... then limit the buffer memory to roughly what you expect to handle and a bit
anything outside of that you can auto-reject
 
nwp
One issue is that you receive full packages only. At 100MB it probably makes sense to process the message as a stream.
 
Is three any BOOST API whose function is almost the same with std::string_view::find?
 
nwp
Probably boost::string_view::find.
 
12:38 PM
You see, there may be two large messages together.
 
nwp
What are the extra 50MB for?
But you know the size of each message, so you read 1 message, send it off to be processed and then read the next.
 
also what's the sensitivity of your processing deadline?
because if it's not that sensitive... I wouldn't worry too much
and if it is... then be willing to buffer and parse multiple things at once and then pause at safe points
 
The faster the better. But no strict deadline is required.
 
so remember in any system there is bandwidth and latency
they rarely go together
latency is solved by small very focused actions
bandwidth relies on moving as much at once as possible
 
nwp, yes, you are right, since I know the exact length. There is no need for the extra 50MB.
Mgetz, thank you for your advice.
 
12:47 PM
FWIW there are safety critical systems that do twin channels for that reason
they have a latency sensitive command and control channel, and a bandwidth sensitive data channel
 
Good idea. Like ftp?
 
or NFS or anything
basically you can send a command like "turn on the machine, or turn off the machine' and expect it to happen. The only reply a simple and tiny ACK
but if you want to query for things that goes in a separate channel where the machine will do it when it can get to it
 
I see. Thank you so much.
One more question arises, one channel for commands which are latency sensitive, the other one is for large data. But both these are using TCP streams. How to guarantee the control channel could be low latency? Only using different TCP ports?
 
dont' use TCP
the command channel is less than a ethernet packet size usually
so you can get away with UDP
the data channel is larger or needs more continuity... so TCP makes sense there
 
I see. It's a choice.
 
12:55 PM
well sort of... TCP has a lot of overhead
so for latency sensitive things it's not great
 
Thank you for your detailed explanation.
 
but for things that don't require a ton of security and are latency sensitive the single packet nature of UDP means you can just sent a packet (encrypted or unencrypted) and reply to the same
 
I see.
 
think about it... when you receive a UDP packet you have the IP of the sender (which you can whitelist to prevent some bad things). So that's who you respond to with an ACK or NACK
for the main data channel you can afford to use something more robust (but slower)
 
Good idea.
 
1:01 PM
so you could use protobuf for the commands if you wanted... and then use something otherwise for the big stuff
or you could even make it simpler
 
I see.
 
1:49 PM
Why this [code snippet](https://coliru.stacked-crooked.com/a/6d122cc4e8550c10) does not compile?

#include<vector>
#include<string>
#include<algorithm>

int main()
{
std::string topic("hi");
std::vector<char> pay_load;

std::fill(topic.begin(), topic.end(), std::back_inserter(pay_load)); //something wrong
}
error: cannot convert 'const std::back_insert_iterator<std::vector<char> >' to 'char' in assignment
912 | *__first = __value;
 
@John because that's not how std::fill works?
 
nwp
Because the third parameter of std::fill wants a value, not an iterator.
 
are you looking for std::copy ?
 
Er, I see. A foolish question.
Thank you.
 

« first day (1806 days earlier)      last day (1052 days later) »