last day (27 days later) » 

00:48
1
Q: How can I parse a large JSON file with repeating values in JavaScript?

BluLotusI am parsing a large JSON file using JSON stream. This works but it returns the file line by line. So when I try and restructure the data I can only get the data that is not repeating. For example, this is the structure: { "Test": { "id": 3454534344334554345434, "details": ...

There are lots of streaming JSON readers... can you clarify which one you're using? Also, what does your actual file look like and what are you trying to retrieve? While JSON does technically support keys with the same name, I've never seen it used in practice.
@BluLotus Do you need to use a streaming parser in this way? Most of the time, line-delimited JSON (NDJSON) is a more appropriate choice. (If you have a lot of records, use NDJSON. If you have big complex structure, you may not be able to.)
jsonformatter.curiousconcept.com Provided this validator accurately implements the RFC7159, it does appear that that standard allows for the duplicate keys. I retract my comments.
@Brad I don't need to use a streaming parser but I've never worked with large files in this way so I'm not sure how else to pull in that data. It is a bit of a complex structure. It was originally XML. Can anyone recommend a good tool for parsing that sort of structure instead?
@BluLotus Without actually seeing your data, it's hard to say. JSONStream is solid, but any parser handling an incomplete document has its tradeoffs, as you have seen. Can you show the code you're using to set up the parser?
@Brad I've added the code, the data essentially follows the above with other repeating values like content
00:48
@Brad , Well, since RFC7159 allows for duplicate keys, JS interpreters only use the last key as a value, all previous values are not usable, they are like comments
@MisterJojo ah! So the only way for this to work is if they were (1) unique or (2) nested in a way I can loop through them?
Any ideas on how I can rename the duplicate keys to be unique: content_1, content-2? I've used text wrangler in the past to manipulate fields with regex but I don't think there is a way to add an incrementer to text with that application.
you must place them in an array of objects like : "content": [ {"id": 123}, {"id": 456}, {"id": 789} ]
@MisterJojo I can't do that manually since there are several thousand entries. Do you have any recommendations on how to go about that more automatically?
you can write your own parser, but it's not easy to do, and not really a clean idea, and not write it in JS but rather in compiled language, because if your file is very big it will cause some problems for an interpreter (just my opinion)
Thanks @MisterJojo creating my own parser is outside the scope of work but thanks for the suggestion.
 
11 hours later…
Tom
Tom
12:02
I made one simple parser by translating Gason from C++ to C#, it would be easy to adjust it in case you are interested - simplest in C# version, possible in C++ too. Also you can compile C# on windows without special environment - .Net is installed almost everywhere => also CSC compiler. There is already a split, iterators and compare functionality implemented in my C# version.
Or you can change it while reading using stream reader - check my 1st question here stackoverflow.com/questions/55372425/…
 
1 hour later…
Tom
Tom
13:21
Here are some small C# programs and batch files Compile.bat to compile them using .Net/C#/csc.exe github.com/eltomjan/ETEhomeTools/tree/master/CSharp

last day (27 days later) »