« first day (3845 days earlier)      last day (1115 days later) » 
01:00 - 20:0020:00 - 00:00

8:01 PM
I have one iterator, Iterator A, that loops through a directory using recursive directory iterator and recursive iterator iterator... then another method that gets the contents of Iterator A and assigns it to a new iterator, Iterator B, to try and access only files of a type (which is found in the contents of the file, this is set in Iterator A)
... this is so difficult to explain coherently and further tells me "I want to light this on fire"
I have to scan through XML files in a directory and import their contents into our system, but I need to import them in a specific order, but the "order" can only be determined by reading a value in an XML file
 
:stare:
 
XD
 
Um. Sounds like you just want to throw a FilterIterator around it (or use a generator) and move on with life.
 
well, I am using it as a generator
 
Or, yes, light it on fire.
 
8:10 PM
try to summarize what I'm doing succinctly but even attempting that requires a decent amount of preface
 
Just pastebin the code so we can share your pain?
 
I can't :/ ... at least I don't think I can
 
Ah
 
the code I'm working on is a perfect case of procedural code shoved into a class
 
@bwoebi ...that is unintuitive from a userland PoV
 
8:21 PM
I'm looping through XML files to import their contents into our system. Iterator A does the looping through XML files. The XML files contain metadata such as the type of file they are: article, image, video, gallery... and so on. Our system has a concept of parent/children objects, so that I can import a parent article that has a child image. Image and video metadata files don't contain info stating if they're a child, but their parent might reference a child's unique identifier
So... I have to import children first before I can import parents, otherwise I can't build the parent/child relationship correctly in our system
at least, that's the conclusion I came to... there were a couple other possible solutions to this problem, but importing children first seemed the easiest
 
is the data set large enough that it can't reasonably all be sucked into memory at once?
 
over 600,000 files, so yes
I run into out-of-memory issues with around 200k :( (which should say more about my code than anything else...)
 
k, well I would probably look at producing some intermediate files which have a my-ns:parent attribute
 
^ yep
 
i.e. resolve the dependencies incrementally and write them to file
 
8:23 PM
normalizing into a DB structure could also be an option
 
then re-parse the files and actually glue all the data together
 
I was going to use an external store
 
doesn't even has to be mysql, could be sqlite
 
@ln-s yeh that's also not a bad shout
 
well... what I have already works, it's just convoluted
 
8:24 PM
In which you just create a coupple of files
 
and I pity the next person who has to work with it
 
if it's convoluted its a pain in the ass to maintain
 
it'll probably be at least a year before this code is ever touched again when I'm finished... these are generally "throwaway parsers"
I'm not saying what I'm doing is good, but I'm too far in and unskilled to completely rewrite what I've done
 
Yeah, a multi-step process seems best at that scale. If you read it all into SQL, it becomes trivial to find anything without a parent and process that, then find anything that is a parent of those and process those, repeat.
 
e.g. I was supposed to be done with this three weeks ago
 
8:28 PM
Beware the sunk cost fallacy. :-)
What's the shortest path from where you are to "it works enough for the next 2 years?" Regardless of what you've already done or not done.
 
^ i.e. if you can get away with doing nothing, consider doing nothing
"consider" being the operative word there
 
Other than leave a comment with an apology to the next developer, aka you in a year, with a recommendation of what to do instead. :-)
 
but in this case I suspect you are over engineering
 
Suggestion: "Get a glass of scotch and a handgun"
 
I added "skipping" functionality so that I could skip past the first 200k already-imported files, but the way how I added it is within two other iterators... and now I want to log when skipping is finished... but the way how I did that instead logs "skipping is finished... proceeding with import" on every, single, subsequent file
@Crell oh, that's already at the top
 
8:30 PM
"Just to think about the shithole you've got into by accepting this job :D"
hahaha
 
If it's just a 2 level tree, and it's a batch process, then time doesn't matter much.
 
I welcome most developers at my team with an image of a glass of scotch and a 9mm
 
@ln-s lol that turned from midday to midnight on a penny :-P
 
"My apologies for anyone who has to use this." and several comments of how I cut corners due to time constraints :X
 
Iterate through the whole tree once, and when you hit a child, import it then drop it from PHP memory and move on.

Then iterate through the whole tree a second time and look for parents, and import them, then drop from PHP memory.
 
8:31 PM
@DaveRandom I truly do that :D
so they can have "fun" !
 
Yeah, it means visiting each file twice, but for a batch process, who cares as long as the memory doesn't blow up.
 
Just one of those projects were management just opens up their a** cheeks to the client, you know
and the technical debt is huge, already to big to address
 
I did not need that visual.
 
I hate myself for being at this job but I'm a money whore too so
and the lead that we have can't even english properly
makes terrible decisions, hacks every piece of code he can (as he is the one who f'ed everything up in the first place by not normalizing data structures)
 
@Crell this @Tiffany, it doesn't really matter how pretty is it as long as it works and it doesn't kill anything
 
8:35 PM
@Tiffany It's all about context. 3 weeks overdue on a long-term project that was under specified or you are under trained for... not so bad! 3 weeks overdue on a project that was supposed to be done in 2 days? Different story.
 
@Tiffany You should see the shit that goes on at my job, we took A YEAR to deliver a feature
A YEAR
RIDICULOUS
 
tbf it depends what "a feature" is
also time is irrelevant here in the 7th circle of Hell
 
Oh not even that complex, just a listing with some filters
 
@Crell roughly what I'm doing... Iterator A is a generator, and yields an SplFileInfo object, then I have a method importChildrenFirst(), assigns Iterator A to a variable then loops over that variable. If a parent is seen in this loop, it continues, and later yields. Within the same method, it re-assigns Iterator A to the same variable, loops over it again, except if a child is seen in this second loop, it continues, then later yields.
 
@DaveRandom outrageous
 
8:38 PM
@Tiffany That... sounds complicated. So you can't tell that something is a child without first visiting the parent to see what it's children are?
 
over engineered piece of crap
 
well, I can generally assume that all images, videos, embedded content are "children"
 
Not even exaggerating
 
@ln-s not helping :/
 
Oh no @Tiffany
I'm not talking about your stuff
:D
 
8:40 PM
well, what I have is overengineered, probably, but I don't need salt rubbed in :P
 
foreach ($file_iterator as $file) {
  if (is_child($file)) {
    import($file);
  }
}

foreach ($file_iterator as $file) {
  if (is_parent($file)) {
    import($file);
  }
}
Would something along those pseudocode lines do the job?
 
@Tiffany was talking about this stupid feature that took a year at my job
if you feel bad about 3 weeks ...
Trying to cheer you up
 
@Crell that's roughly what I have, except what I have has to be a generator because it's fed into something else that does the actual importing... though at one point I did consider rewriting to do what you did...
 
So the import routine needs a generator, not an item to import?
 
because it would be far simpler, but I don't know if I can justify spending a day (or possibly more) doing that
 
8:42 PM
It sounds to me that you are already knee deep @Tiffany try to simplify it
just branch off to another branch
leave the current work as it is, if you see that you can't do it then go back to your previous branch
 
(Of course, I'd probably make is_child() a wrapper iterator filter, possibly using a generator, and then use a map operation over it on import(). But then I'm that kind of guy...
 
@Crell trying to think of how to explain it... the class I'm writing is ran through a domain-specific command, which needs an end condition
coworker suggested using an iterator built around the file directory to make it simpler for supplying an end condition
 
I guess I don't get why the iterator/generator/loop needs to leak into the import process at all. The importer should be given a single item, do its thing, and return. If there is no "import this one object" operation, then that's the bug you're fighting against right there.
 
there are a couple of methods that I have to override from the parent class where I just have return $this->oIterator->valid();
 
Fix that and the rest becomes a lot simpler.
 
8:46 PM
@Crell there is a method that does importing, but it's inherited from the parent class
and I'm overriding it
I'm using the generator to work within the confines of what the parent class and/or command-I-run-to-do-the-importing is expecting
importChildrenFirst doesn't do the importing, though now I see where confusion comes from with my naming... importChildrenFirst is a generator
 
I would separate the process in tasks:

Find the files
Parse the files, parsing the files returns a parsed model structure
Import the parsed structure
 
but it's a generator that also takes in another generator
 
Possibly derive to other more specific importers, for example, ImageImporter implements ImporterInterface, VideoImporter implements ImporterInterface and so on
 
yeah, we have several already like that
 
Perhaps the main importer, it's just a collection of importers
 
8:51 PM
probably a couple hundred x_x
 
If you are using drupal, I think that uses the symfony DIC, so you could easily tag those importer services and collect them
 
@Crell missed this, but <3
 
It really sounds like "using the existing buggy importer routine" is the root problem here. :-)
 
that plus separation of concerns / logic
 
nope, it's an in-house CMS, built by some of my superiors :P though a coworker is working on importing data from drupal as well
 
8:53 PM
oh
 
Here's a question: Would the existing importer handle it better if you could pass in a single iterator of just the child objects, let it do its thing, and the end-case is "the iterator is empty"?
 
@Crell yeah... I agree with this... but it also provides a lot of scaffolding I can rely upon, that I otherwise would have to write myself, and I am not skilled enough
 
I insist there must be something which finds these xml files and then foreach found file a parser must be called, this parser could return or yield a structure, foreach yielded structure the importer must be called
 
at some point I will have to find out what the rules are for code sharing
it would be so much easier to throw this into a gist because there are some parts I'm rather happy about, but most of it is a mess
 
Sounds like most projects I've worked on... :-)
 
8:59 PM
@Crell I'm not sure that it could handle it... at that point I may be better off instantiating the scaffolding for some of the core pieces to set up the relationship myself rather than relying upon the parent class
which I have seen done in parsers before, but I've been trying to avoid doing it
 
Good day all
 
parent class and/or command-I-run-to-do-the-importing
 
It's hard to give more specific advice without seeing code. In general, though, I agree that reassigning an iterator to an iterator like you were describing is a code smell. Instead, focus on giving the importer what it wants to do it its way, then accept that you'll just make multiple passes over the file tree and call the importer multiple times. That way you don't need to build any 100k item lists in memory. The runtime will be longer, but it's a once a year batch process so who cares.
 
I see now how I could do that... basically means running the command more than once, but I'm already doing that to deal with OOM issues
and I'll have to do it anyway
 
@MarkR Hello there, whatever time zone you are in
 
9:07 PM
he is SMT (Sheffield Mean Time)
it's like GMT but "meaner", it calls you names and stuff
 
lol
 
in a yorkshire accent
 
Meaner would be shitffield ? :D
No offense intended
Just making a meaner pun
 
the main the thing you need know about Sheffield is that literally everywhere is up a hill from everywhere else
 
Ehh it's 10 o clock there but ok
 
9:09 PM
especially when you have been drinking
 
22 hours to be precise
 
22:10 by Manchester time
 
exactly
I suppose minutes a common thing in every TZ
 
@DaveRandom Manchester isn't?
 
sorry I am really excited because I somehow forgot this song existed since ~1998 open.spotify.com/track/…
@Tiffany no manchester is in a bowl of hills but largely flat in the middle
that's the main reason it's here, the bowl traps moisture coming in off the Atlantic and causes it be generally very humid/rainy
which is good for spinning cotton, which is why this city exists
 
cmb
9:14 PM
@ln-s Asia/Tehran is off by 30 minutes
 
also a popular fact that is that until the mid C19th, Paris was 11 (17?) minutes ahead of London
 
@Girgias apparently he doesn't understand "write in your own language" - he attempted once more... gist.github.com/Danack/4d1480a262e2cb62c6ff4cbf49fa7aa1 I think I'm not going to persist any further.
 
9:41 PM
@DaveRandom wow... why?
 
@StatikStasis Because they disagreed on where the Prime Meridian should be.
England thought it should run through London. France thought it should go through Paris.
 
I must have missed that day when this was discussed in class.
 
well, that's just after they tried to create a standard meridian
 
I've heard Derick teach this course.... several times.
 
before that, the time would be different because, well, the time is different there
time zones aren't a natural phenomenon
now we agree on the meridian, but Spain is stranded in completely the wrong time zone because Franco wanted to be Hitler's buddy
 
9:47 PM
From wiki: "The French clung to the Paris meridian as a rival to Greenwich until 1911 for timekeeping purposes and 1914 for navigation. To this day, French cartographers continue to indicate the Paris meridian on some maps."
 
the now-agreed meridian runs to the coast not far from here
also through the town where my parents lived; they used to cross from the Eastern to Western hemisphere to do their laundry
 
"The United States passed an Act of Congress on 3 August 1882, authorizing President Chester A. Arthur to call an international conference to fix on a common prime meridian for time and longitude throughout the world." So... looks like it was America to the rescue again... =D
Sorry- I had to...=P
 
U! S! A!
 
begins chanting
 
I seem to remember the US had their own proposal, but got out-voted
 
9:54 PM
'MURICA!!!!!!!!!!!
COMIN' AROUND TO SAVE THE MOTHER FUCKIN' DAY, YEAH!!!!
 
Paris and London had the advantage of putting the date line in the middle of the pacific
 
Oh, I'm sure the US wanted D.C. to be the meridian and Europe was like, "lol... no."
 
@IMSoP mere details...
 
Yeah, the antemeridian is well positioned.
 
Putting it somewhere in the mid-Atlantic would have made more sense.
 
9:55 PM
Huh... Mars is going to have fun with that problem....
 
Even just off the Irish coast would have worked, and still kept the antemeridian in the Pacific.
 
Even if it's terraformed to have an ocean again, there's still going to be a contiguous landmass around the planet.
I'll put off thinking about that problem until it's relevant.... centuries from now (at best)
 
@Crell that would have been too far to go with the laundry
 
10:12 PM
@Sara lol I don't even think I have seen that, I am just that dull that I know it anyway :-P
it probably came from QI tbf
 
@StatikStasis "That the Conference expresses the hope that the technical studies designed to regulate and extend the application of the decimal system to the division of angular space and of time shall be resumed" - still waiting on that one.
 
@Sara So are you saying that we should nuke mars the shit out of it ?
 
It's a good thing they didn't. The use of non-highly composite numbers as base values is a terrible idea. The French got it wrong, once again.
 
Elon said it
 
/me votes we switch to base 12, and use 360 as the base value for all common quantities.
Or, decimal 360. Which in base 12 is 260.
 
10:16 PM
....I don't even think that's a bad idea
I really like the idea of base 12 currency at the very least
 
@DaveRandom It isn't :-)
 
I would like an intermediate 60 unit between 12 and 360
 
Depends on the situation. 60 can be considered.
 
time and money are the two main places where 360 is too big a number to divide quickly in your (my) head
 
@DaveRandom I also miss crowns
 
10:21 PM
In mathematics, a superior highly composite number is a natural number which has more divisors than any other number scaled relative to some positive power of the number itself. It is a stronger restriction than that of a highly composite number, which is defined as having more divisors than any smaller positive integer. The first 10 superior highly composite numbers and their factorization are listed. For a superior highly composite number n there exists a positive real number ε such that for all natural numbers k smaller than n we have...
 
@Danack I miss copy-pasteable URLs from wikipedia - oh no I don't, they never existed
 
240 is a highly composite number, but not superior. But that's why it was 240 pence to the pound in the UK, and in much of of Europe, from Charlemagne onward.
 
360 is a good unit I am sold on that, I just want a lower "standard" intersection with decimal I guess (which is not useless)
 
@Danack LOLOL! I have some bad news... I don't expect a change anytime soon. =/
 
also I can count to 60 without getting distracted, much less so with 360
 
10:25 PM
60 and 120 are also superior highly composite numbers, so can be used where appropriate.
 
1) shut up :-P
2) try it.
 
But as someone who does metal working and woodworking, trying to measure things in inch-fractions is utterly maddening and moronically stupid.
 
I talk in feet and inches, but I measure things in mm
 
I should probably just get some mm-marked tools and say fuck it.
 
wait wut you don't have those?
 
10:29 PM
I live in a 3rd world country, remember?
 
i dunno I feel like I live in a weird twilight zone there
the world I live in works in metric, but "workmen" "talk" in imperial
i.e. tradespeople tend to casually express things in feet and inches
and I even use yards in casual speak
but if was going to measure a thing I would do it in metric
I suspect that is a very narrow age group to which I belong :-P
 
Age and geographic group.
 
lol true :-P
use metric for measuring physical things kids, it's much easier
vs dozimal for "abstract concepts"? seems like a worrying sensible off the cuff remark
 
Were it possible to switch, dozenal would make more sense for measuring physical things. Easier to divide into halves, quarters, thirds, sixths, etc.
 
I'm not against it, it seems logistically impractical to make the world change in a timescale I could witness though
which is not to say I wouldn't be willing to help make it happen, but it makes me know much less what to do about it
also, solid work on seamless dozimal/dozenal spelling correction there :-P
 
10:47 PM
lol
 
@Crell You just haven't done it enough. That's probably the only area that I do like inches and feet is in woodworking. But it's only because I have been doing it my whole life so I am used to it.
 
Measuring a fraction of an inch on something, or finding a tool that is the right size in that measurement, is batshit insane.
 
1/16, 2/16 or 1/8, 3/16, 4/16 or 2/8... you just get used to it after a while.
 
Is a 5/7th nut bigger or smaller than a 3/4th?
You also get used to abusive spouses, but that doesn't make them something to keep around.
 
Oh yeah... lol that is aggravating. I prefer using metric tools. It is much easier.
@Crell Is... is everything okay? =P
 
10:53 PM
LOL. In that department, yes. :-)
 
I hate when a nut is actually standard rather than metric and I use a metric socket that provides just enough variance for me to strip it.
It doesn't happen too often, but when it does it's frustrating.
@Crell I do have an app called FractionPlus I use often when doing woodworking and I need to measure some values that would require me to figure out the common denominator and reduce... it's just quicker to use the app.
 
@StatikStasis indeed, the actual problem:
 
I know. Standardization is nice when feasible. Getting people to agree is hard.
 
I only read "Standardization is nice. people is hard."
"people when feasible"
 
Same thing. =P
 
11:45 PM
@DaveRandom oh god, I was talking about the roundabout. English isn't my first language, I speak Americanish. =oP
 
:-P <3
the rest of that message was just so perfect I couldn't resist, sorry
 
01:00 - 20:0020:00 - 00:00

« first day (3845 days earlier)      last day (1115 days later) »