« first day (176 days earlier)      last day (3348 days later) » 

12:01 AM
@TeaDrivenDev I found myself really liking MailboxProcessor but it is a really slow solution
 
So, I need to scan a filesystem. First simply collect all the file names, then parse each of the files. That may be thousands of files in at least tens, maybe hundreds of gigabytes.
 
what type of information are you trying to extract?
and is there a reason you need to pull out the filesname in advance?
 
Mainly for being able to display some kind of progress.
 
So it's a MapReduce problem, this blog post might be a good place to start bartoszsypytkowski.com/blog/2014/07/09/fsharp-akka-map-reduce it uses akka.net but the concepts are the same
 
I'll explain the solution I thought of to see if it makes sense at all.
Parsing files is of course easily parallelizable, but as the task is heavily I/O bound, "automatic" parallelization say using Async will like not choose the most efficient degree of parallelism.
(I tried that for a handful of files, and it was slower than sequential processing.)
So I tried to think of a way to control the number of "threads" myself.
One MailboxProcessor would "race ahead" and simply collect the filenames to drop them into another one's queue.
This next one would have a "pool" of further agents to dispatch the individual files to, monitor the overall throughput and try to optimize it by experimentally increasing/decreasing the number of active "scanning threads".
 
12:14 AM
I'd suggest having a master which works to stream through files and then a number of workers, each of which has your mapper function. These then pass messages through to a collector or a group of collectors. This then spits out the result once all of your files have been processed
 
in general, for file IO, you're better off with one thread, unless the files sit on different disks
(or, of course, there's heavy CPU processing and not just 'scanning' in volved)
so simple and straightforward may be the best...
(that's if you're just scanning through the files, and not having to do much/any "work" on the contents)
 
I'd recommend the original MapReduce paper static.googleusercontent.com/media/research.google.com/en//… it explains a lot of the concepts which you're looking to do if you definitely want to solve it with MailboxProcessors plus if you were to then use something like FSharp.CloudAgent you could easily distribute it across a bunch of worker processes if you needed to
 
I don't know how much work exactly is being done on the file contents, as that's handled by a third party library, but I think it essentially comes down to collecting some numbers.
Thanks @bruinbrown; I don't "definitely" want to solve it with MailboxProcessors; I want to solve it in the most time efficient way. :-)
 
@TeaDrivenDev if you just scan a bunch of files in a loop - and a CPU isn't spiked/hammered, parallelization will probably not help (unless you can move to multiple systems or at least multiple physical drives) - might be worth checking before you engineer a solution that doesn't actually improve things ;)
 
@ReedCopsey I just thought of parallelization because there's a file system sync tool that I use (FreeFileSync) that shows how many scanning threads it's using while looking for changes, and that number is usually around 16 on my system.
 
12:23 AM
it really just depends on the amount of work being done
if you're component does real processing (other than just reading and comparing something) at all - then it's worth it
 
I'd say that MailboxProcessors would be a solid option here if you're doing a lot of work otherwise it's not really worth it
 
if not, it tends to slow things down
 
See, that's the problem with programming
There's too much stuff you need to read before you can actually do things. :-)
 
Heh, it's funny you say that today
 
12:29 AM
Why?
 
Somebody in the distributed systems community on Twitter said that there's no point reading any past literature and you should just work on stuff
 
I wish that were true.
I'm too impatient for this reading thing.
 
I'm starting to prefer reading papers over blogs but they're so hard to discover without paying through the nose
Like the active patterns paper cleared up so much more about them than any blog post I found did
 
active patterns paper? @bruinbrown
 
12:55 AM
My F# "architecture" always reminds me of writing JavaScript; a bunch of functions in a bunch of files, but not much of a concept to it.
 
 
7 hours later…
7:45 AM
@TeaDrivenDev I would also encourage you to look at RamDisk products - sometimes they help more than any optimization on your end - and I would also have only one thread which reads and then a worker crew accessing the mailbox processor - have not done this with F#, but more with C# and BlockingCollection
 
 
5 hours later…
12:30 PM
is the mailbox processor deal thread safe?
I used it because I thought that's one of the main things it was about, but it's early in the morning and I'm wondering
 
@Maslow thread safe as in?
 
as in able to be called from multiple threads without the possible of side-effects
 
Yes. As far as I am aware it has a concurrent queue (that is internal to it).
Will be back in a bit. Home time now.
 
Single Reader Multipler Writer according to MSDN
 
 
1 hour later…
1:39 PM
|>
So, by now I have determined that a single reading thread should indeed be the fastest option; at least as far as spinning rust is concerned.
 
1:51 PM
With an SSD, using three or four threads shows significant improvements; there seems to be no benefit (but apparently also no significant penalty) to using more.
 
@TeaDrivenDev I would try a RAM disk as well - depends on your workflow
 
It's not really feasible; this is about an "end user" desktop application.
But I think I got your earlier comment now - having one thread that does nothing but read the files straight into memory and then hands them off to others for (even light) processing seems to make sense as an optimization.
 
it is what you do when you deal with high load network operations as well
 
 
1 hour later…
3:04 PM
@TeaDrivenDev did you find anything good about Caliburn Micro?
 
In general? They have quite extensive documentation, but it's a lot to go through when you're trying to solve a specific problem.
 
3:23 PM
I don't suppose there exists a CMS for F# mvc ?
 
3:41 PM
I like this backward pipe operator no more parens!
 
4:31 PM
@Maslow Not sure about the actual operator, but that is nice, yes.
 
You know, I bet in a language like F# where every block of code is so small and insular, source control merge conflicts basically disappear completely or are unbelievably easy to handle...
Anybody can say of such?
 
I'd guess that's the case
 
I have no practical experience with that, but I'd think so too. Certainly easier than in C#.
 
5:03 PM
It really dependes
Overall, it's quite improved
Most of the benefits in reasoning about the consequences from a merge fall out of using immutability both globally and locally
But there are still the odd times where there is local mutation and two diffs touch one another. It's rare, but happens. Oh well, just put your work shirt on :)
And a comprehensive refactoring will get hairy in a merge regardless of the niceness of the code.
 
5:17 PM
@BryanEdds my thoughts were around the size-of-individual-pieces being reduced would lead to less conflicts and easier to resolve ones (partially due to free functions, partially due to conciseness in the language, largely due to looser coupling allowing changes to be more isolated)
just a hypothesis
 
You're right, I'm just saying there's a limit to how comprehensive that conclusion is in practice
 
@BryanEdds supposin' so. Just popped in my head when I was walking through the office this morning and heard somebody saying to their manager "...there were a bunch of merge conflicts, so I might have screwed something up" and I know this is a totally acceptable fact of our jobs, but it occurred to me- should this type of mistake be so prevalent as it is? Perhaps it shouldn't...
 
you're definitely right
 
5:54 PM
@BryanEdds any response to the F# unreal thing I linked to you a day or two ago in here?
 
I'm not sure what you were even asking?
 
the thing about either unity or unreal supporting F#
 
Oh, no, that's not mine. That's something Miguel did.
deicaza
 
ts
Always blaming others for something you didn't do.
 
story o my life
 
6:23 PM
@TeaDrivenDev, if you're having trouble achieving coherency in the organization of your code in F#, try to organize it around your types. If there are not enough types, perhaps implement more of them. Modules are another way help package your functions up with the types to which they are related.
In brief, organize your F# code around types!
 
7:09 PM
@BryanEdds you have an account on project euler?
or anyone else in here that has one.
 
 
1 hour later…
8:27 PM
I'm F# instead of Haskell because I don't want to have to invent a new branch of computer science every time I get stuck :)
 
8:40 PM
that was a great comment in twitter
 
@BryanEdds pfft have you learned Haskell? The last thing we need in the FP community is flinging crap at eachother's good ideas which are frankly all head and shoulders above what the industry has available now... you just aid the OO folks in discrediting FP with comments like that
 
shut up jimmy
 
haha
IMO every FP language I've touched has been miles ahead of the enterprise garbage most all of us are stuck in which is basically C# and .NET 2...
 
9:01 PM
@BryanEdds I'm trying to do that; it's still quite chaotic, though.
 
just remember to stick to only global lets and static members where possible
once you start using instance methods in F#, you get pulled back into the confusing mess of OO
and it can be hard to extricate yourself back out since OO has a sort of infectious property to it
sorry, I meant "stick to module-level lets", not global lets
 
9:51 PM
Vote here for F# support in ASP.Net vNext - aspnet.uservoice.com/forums/252111-asp-net-vnext/suggestions/…
 
@BryanEdds A problem I have with module-level lets is handling global "dependencies" like a database context that is set up once and then remains unchanged. I have functions in modules that need it but don't want to pass it around all the time.
 
you have to give up on that
you need to use dependency injection style to get good functional code organization
if you find yourself having to pass around multiple dependencies to each function, factor out a single 'master' dependency type
that includes all the dependency references you will need
then you only have to pass around one master dependency ref
from which you can access all others that you may need
it is not nearly as inconvenient as you might think if you do it that way
I usually have a World or Env type that contains two things -
1) the full state of the program, and 2) the dependencies that are used throughout the code
This is typical in functional style from my experience
 
10:11 PM
Nice - F# support in asp.net vnext is now the #2 top request on user voice :)
 
and I had not voted, +3 now
Friend vote, unlikely that I will write any asp
 
and it's #1 :D
 
Well, that escalated quickly.
 
the F# community is awesome ;)
 
I'll spam it in the c++ room, not sure how many of the linux hipsters in there who write F#
 
10:24 PM
they'll be all like "OMG, M$$$$ lollsszz!"
 
@JohanLarsson I heard they're called "heapsters" now. :D
 
:)
The votes ++ed right?
 
How long will a MailboxProcessor wait on Receive()?
Will it ever time out?
 
10:47 PM
not unless you specify a timeout
but if you don't put one it, it waits forever
 
Damn! @ReedCopsey beat me to the punch
Gotta type faster
:-D
 
@pblasucci well, I'm going to go eat something - so you've got a good half hour to get a chance to beat me to the punch ;)
 
hahaha
NIce
I'll take what I can get
 

« first day (176 days earlier)      last day (3348 days later) »