« first day (2220 days earlier)      last day (629 days later) » 

Ron
10:20 AM
My bad. Please discard.
 
 
5 hours later…
3:12 PM
This probably doesn't deserve it's own question.
Is there a nice C++ library out there that provides a "dense" hash map without all the overhead of one specifically meant to be persisted to disk?
Conceptually just a std::vector<...> with a binary search function, but wrapped in a nice interface.
(The difference being that open addressed implementations are not necessarily "dense" in that they will resize to have extra space. Whereas in my case the number of elements does not change after initialisation and as a bonus I can provide a perfect hash function.)
 
Thanks, will look at it. I am targeting C++14 or C++17 unforutnetly.
 
umm that's since C++_98_
it's ancient
still works though
 
That's odd, the link says since C++20
Oh, n/m, misread it
Was made constexpr in C++20
 
3:50 PM
Is there a fundamental difference between UINT64_C(1729) and uint64_t{1729}? I assume UINT64_C is only useful in C and C++ before uniform initialization.
 
nwp
4:01 PM
If you can improve your perfect hash functions to produce consecutive numbers starting from 0 you can use an array.
 
4:43 PM
1 message moved from Lounge<C++>
@HenriMenke well UINT64_C isn't a standard type so hard to say?
 
How do you guys typically cut a tcp stream into chunks?
If I am not mistaken in UDP you can, when reading a socket, just read a full datagram. But in TCP there is no such thing you just have an incoming stream of data
do you terminate every packet you send out with a special character or something?
 
nwp
You use protobuf or similar.
 
And if you weren't able to use protobuf?
 
nwp
Then you reimplement protobuf.
 
sounds a lot of fun...
 
nwp
4:54 PM
Though what I said isn't quite right. Protobuf doesn't have a way to figure out where a message starts and ends. You have to do that manually.
 
aha!
And how do you typically do that?
 
nwp
2 bytes for the size at the start of the message for example. Alternatively 1 byte for the message type. Once you know the type then protobuf will figure out where to stop.
 
That's what I thought as well but...
lets assume that you send 2 individual TCP "packets". The first packet arrives, it is 100 bytes but for some reason there was a bug and you only receive 50 bytes. This means you will start reading 50 bytes of that second packet, which will lead you to incorrectly interprete the data
 
nwp
At work we did something extra. \FFFFFF denotes the start of a message and there is additional logic that handles messages containing that. The purpose is that you don't need the state of the stream and can find the start of a message. You could do something similar.
 
I was thinking about something very similar indeed, some sort of magic number or whatever. but that sounded just too stupid which is why I thought there must be a better way
 
nwp
4:58 PM
Well, by "we" I mean someone else. I didn't actually do that. I'm not a fan, but I have nothing better either besides "Just don't put yourself into a situation where you need that".
I wouldn't try to defend against my TCP implementation being bugged and losing data. That just doesn't happen. TCP is specifically made for this not to happen. Fix the bug, don't cover it up.
 
@LandonZeKepitelOfGreytBritn I mean what are you trying to do?
 
Trying to anticipate such a potential issue:
3 mins ago, by LandonZeKepitelOfGreytBritn
lets assume that you send 2 individual TCP "packets". The first packet arrives, it is 100 bytes but for some reason there was a bug and you only receive 50 bytes. This means you will start reading 50 bytes of that second packet, which will lead you to incorrectly interprete the data
 
ah ok... so do you have two bytes that can't appear in the interrim data?
or can you send in text?
Because either you have to send a length header and have a pre-agreed max packet length
ooor you need to do something like base64 encoding and have a CRLF terminator
 
yes, I understand what you say and agree but
 
FWIW the length header thing is why jumbo frames aren't a regularly used thing
 
nwp
5:03 PM
The length header and max length are not actually enough. You can detect the error, but not recover from it.
 
well yes.. CRC is also needed
 
nwp
Assuming a continuous stream of bytes you would still not know what to throw away and where to continue reading.
 
when for instance using the write syscall to write on a socket like below. Can you easily append such a thing?
 
also the whole you only receive X bytes thing can be a kernel issue on the receiving system
 
typedef struct {int foo;} bar_t;

bar_r bar = {123};

...
write(myTCPSocketFD, &bar, sizeof bar);
...
 
5:04 PM
are you setting socket nowait?
have you set your iocntls correctly?
turned off nagels
 
no I am not setting socket nowait. It will just block and read.
 
because Nagle's is a killer
39
A: How would one disable Nagle's algorithm in Linux?

SoravuxThis flag (TCP_NODELAY) is an option that can be enabled on a per-socket basis and is applied when you create a TCP socket. This is done for a purpose: Nagle's algorithm is generally useful and helps handle network congestion. I doubt you want to disable it system-wide since your system will prob...

 
reading this as we speak :)
 
So Nagle's will kill you on the Sending side
but you shouldn't rely on it not being enabled
 
OK yes, but this (disabling Nagle) does not solve the issue
 
nwp
5:07 PM
@LandonZeKepitelOfGreytBritn It depends on what you consider easy. It's not that difficult to implement but somewhat tedious and easy to miss a small bug. Just wait for the preamble and run the preamble-escaping logic.
 
you should assume you have garbage coming in
 
nwp
*unescaping
 
but honestly nwp has the right of it... just implement protobuf if you can
 
Apparently implementing protobuf does not solve this issue either, as it doesnt know where to stop
 
again.. what are you trying to send?
 
5:09 PM
structures of random sizes, containing strings, ints and whatever not
 
Because all your correct fixes mean encapsulation
then you need to have some way of telling the receiver what to expect, that needs to be part of the packet
if I can get struct A B or C but there is no common header field then how the hell do I know what I'm recieving
remember Internet Protocol (IP) is inherently unreliable, that's why TCP exists
 
nwp
@LandonZeKepitelOfGreytBritn It does know where to stop. The problem is to know what type of message you have. And you can use a byte for that.
 
How does it know where to stop?
 
nwp
Because it knows the message type and that contains the information how to parse it.
 
Unless you provide it with a "post"-amble that it has to detect as well
 
5:12 PM
well if you're sending binary data you need to send it with a length. But you also need to have a maximum message length
after which it gets split
any sort of system to system protocol is generally speaking documented to the 11s
 
@nwp Yhea ofcourse but what if half of your packet goes missing? You ll start reading the second packet if there is no pre/post amble
 
because there can't be any ambiguity
 
nwp
Internally protobuf uses length bytes with some trickery to save memory.
 
@LandonZeKepitelOfGreytBritn you're making the fundamentally incorrect assumption that packets won't be delivered together
that only holds true for UDP, and even then only sort of
for TCP you need to assume a stream
 
indeed that is what I am assuming here. Well.. I am assuming this could happen due to a bug or a hardware issue somehow
 
5:14 PM
it's not a bug
it's a design feature
 
it could happen because your reciever is doing something else or your switch is being efficient
 
Imagine if your length field gets corrupted. That s really not far fetched IMO...
 
nwp
@LandonZeKepitelOfGreytBritn Ah, protobuf cannot handle that. Either it's accidentally a correct message and gives you the wrong result or it gives you a parsing error. The incorrect message you can detect via a checksum and the parsing error you recover from by throwing away the bytes until the next preamble. And you need to notify the other side to resend packages somehow.
 
then you drop the packet
 
5:15 PM
But how will you know you have to drop it? length just got changed from 10 to 20 bytes
 
@LandonZeKepitelOfGreytBritn or is deliberately set wrong by a malicious actor?
 
doesnt matter IMO, result/problem is the same
 
@LandonZeKepitelOfGreytBritn Let's say you have a message header that indicates type, length, and checksum. You check those before parsing. You'd also have a maximum possible length.
 
@LandonZeKepitelOfGreytBritn At least in my opinion, you have two reasonable options: either you trust that the implementation of TCP you're using is reasonably free of glaring bugs (in which case this is completely unnecessary) or else you don't, in which case it's better to not use it at all. In the latter case, if you want TCP-like behavior, you can implement it yourself on top of UDP. There are quite a few libraries around to do exactly that, so choose one and roll with it.
 
nwp
@LandonZeKepitelOfGreytBritn That is likely to cause a timeout where it waits for an additional 10 bytes that never arrive or contain a preamble.
 
5:17 PM
@nwp Aha! I think this is indeed the solution
@Mgetz and this as well
 
TBF this is kinda basic security too
 
nwp
We did this for Bluetooth connections. It turned out to be useless in the end though. The initial protocol errors were caused by the device hardfaulting and restarting semi-randomly, resetting the state. Once that was fixed Bluetooth turned out to be surprisingly reliable, contrary to people complaining about it to not be.
 
Add a length field, preamble and checksum. Read the number of bytes and verify the CRC. Break the read operation whenever you encounter the preamble or all bytes are read.
If you read part of the "second" packet (after having read the first part of it because half of the first packet was missing) your CRC will be incorrect so you will discard it. You will then proceed on reading on your socket again (an unknown/incorrect) number of bytes. Those read bytes will be absolute garbage all the time untill you reach the preamble. WHenever you reach it you know you are back on track...
Hopefully this makes sense.... To me it does
 
nwp
The length field could be part of protobuf.
 
But then, I am wondering how you add a preamble to this:
20 mins ago, by LandonZeKepitelOfGreytBritn
typedef struct {int foo;} bar_t;

bar_r bar = {123};

...
write(myTCPSocketFD, &bar, sizeof bar);
...
just add a field which always has the value of that preamble?
#define PREAMBLE 123
typedef struct {int preamble; int foo;} bar_t;

bar_r bar = {PREAMBLE, 4455};
 
nwp
5:25 PM
You do that outside of the data.
 
Yhea ofcourse, my bad
I just put it in a single structure for the sake of simplicity. You d typically have some nested structure(s) here
 
nwp
PREAMBLE + escape_preamble(encode(bar));
encode is done for you by protobuf.
I guess you should add + CRC(encode(bar)) to the end.
 
put the CRC in the preamble
 
in the header or preamble?
header makes absolute sense, not sure about putting it in the preamble...
 
@LandonZeKepitelOfGreytBritn In the header. If you're going to re-create TCP on top of TCP, you might as well do it the same way TCP already does it, right?
 
5:30 PM
preamble, header+CRC
 
@JerryCoffin Absolutely
 
nwp
Maybe you can just nest TCP implementations 🤡
 
@LandonZeKepitelOfGreytBritn That was sarcasm. If your hardware (or whatever) is so broken that the underlying implementation of TCP doesn't work at all, then what makes you think your doing the same thing over again is going to make it any better? If you honestly have something so broken you can't rely on TCP, then you probably need something like forward error correction to do any real good.
On the other hand, if what you need is really just framed TCP, just use websockets.
 
6:06 PM
@Mgetz Of course UINT64_C is standard. eel.is/c++draft/cstdint.syn
 
never knew that existed
 
nwp
Me neither, but it makes for cool trivia.
 
I mean technically I'm still correct... it's not a type ;p
it's standard macro
 

« first day (2220 days earlier)      last day (629 days later) »