« first day (3742 days earlier)      last day (1206 days later) » 

12:49 AM
@traducerad 0% if there's no collisions
Poor sod: superuser.com/q/1616449/53088 A coworker once tried this at 9PM Friday night, the anguished lad was in tears til Sunday noon.
 
 
6 hours later…
6:46 AM
 
 
4 hours later…
nwp
10:46 AM
@traducerad All it takes is someone using a FIFO instead of a LIFO somewhere because it's easier to implement.
 
11:41 AM
or 2 different paths because load balancing decided by a flipping bit
 
 
4 hours later…
3:37 PM
Good morning
 
4:06 PM
hi
 
 
2 hours later…
6:27 PM
Good evening, hi.
So what's software development gonna look like 100 years from now? Is the job "software developer" still going to exist? Will AI replace us? Or is programming going to be a kind of activity that is totally different from what were doing now (like in Minority Report)?
 
6:44 PM
butlerian jihad will make software engineering a criminal offense akin to necromancy in contemporary societies
So threading, shared pointers, and asking the candidate literally anything about building code were considered too specialized of coding questions. But dynamic programing. That stuff is apparently key to a successful software career.
Ugg, I think kept pissing people off with my opinions that C++ engineers should know how to write a producer consumer queue.
 
7:16 PM
@StackedCrooked Given that computers haven't even existed for 100 years, I doubt we have enough data to make a meaningful prediction. That said, from my perspective over at least the last 20 years or so, change (and especially, improvement) has been distressingly minimal and slow.
 
There wasn't source control 30 years ago?
 
@JerryCoffin I was thinking the same. There hasn't been a major breakthrough in the last 20 or even 30 years.
On the on other hand, there have been breakthroughs in AI in the last decade. So maybe that will change things...
 
No code solutions are becoming increasingly more viable. Even in the data analysis space.
 
But then again, if you look more closely, AlphaGo which beat the best Go player in the world wasn't even a neural network. It worked similarly to a chess engine but with an enormous amount of cloud computing behind it.
So maybe the AI advancements aren't as big as they seem. (Feel free to correct me. I surely WANT to be corrected on this.)
 
AlphaGo and its successors use a Monte Carlo tree search algorithm to find its moves based on knowledge previously acquired by machine learning, specifically by an artificial neural network (a deep learning method) by extensive training, both from human and computer play.[4]
So objectively wrong
Not to mention it has transformed my old field (medical imaging) because it's much easier to do object recognition.
AI has been used to generate "variants" of designs (3d models, websites, GUI for applications).
AI has been used (experimentally) to automate and detect common tasks for CAD programs (better shortcuts in blender). Not to mention updated versions of intellisense.
The generation of variants is particularly important for video game asset design.
Would be cool to make a make a "bad commit" detector
 
7:38 PM
@StackedCrooked AI advancements have mostly come down to faster hardware. I did some work on artificial neural networks on the 1980s, but training it on a 20 MHz 386 quickly became impractical for anything but truly tiny models, so it remained purely proof of concept until hardware caught up.
 
^ LOL. AI advancements aren't due to faster hardware at all.
Many optimization techniques documented in the 1970s/1980s weren't functional past a few hundred elements because they weren't correctly regularized.
 
I think applying ML in software engineering can probably be fruitful in some areas (essentially improving lint-like tools). I doubt it renders software engineers obsolete though--it just makes an ever higher percentage of the job oriented toward figuring out what users need/want (which is something they themselves almost never know, so if you just take their word for it, you'll get things almost completely wrong).
 
In short the 2000s isn't wasn't possible to train the kind of networks we use today to convergence.
 
@Mikhail I'm basing myself on the explanation given in the AlphaGo documentary on YouTube. (The explanation was given by the creators by the AlphaGo team.) I'll try to find it and post it here.
 
I'm too lazy to argue or watch a video :-)
 
7:42 PM
I'll link the exact timestamp.
 
@Mikhail Try to train a model in a 386, then come back and try to say that with a straight face.
 
^ I don't really understand the first component. But it's clear that component 2 and 3 are not using a neural network.
 
@JerryCoffin Foremost people were training them, and using them even earlier (1980s).
It wasn't a good strategy from an accuracy perspective.
In the early 2000s people tried training much bigger ones on super computers etc with somewhat mixed results (better than before but still with accuracy issues)
 
@Mikhail How do you figure 1980s is "earlier"? The 386 came out in 1985. If I didn't know better, I'd almost think you were actually trying to support what I said by appearing to argue against it, but then ensuring that everything you said was obviously false.
 
Like 1982
Anyways, perhaps our divergence of opinions comes from my experience pushing accuracy of models (or optimization). I think if you worked on the algorithm/accuracy side of things you'd think accuracy/algorithm was critical :-)
That being said, I have never really encountered a problem of computing resources helping convergence.
 
7:59 PM
@Mikhail Hey man don't treat me so harshly. I certainly believe in AI! I don't believe human intelligence is beyond what computers can achieve, because that would mean human intelligence is somehow supernatural. I don't believe we are supernatural, hence I believe true AI (smarter than humans) must be possible. But I also am critical of today's achievements and I believe I should always play devil's advocate every time someone claims a breakthrough has been made.
 
@Mikhail Maybe it comes from the fact that I have actual experience from the time frame under discussion, and you don't?
 
@JerryCoffin I mean the algorithms were invented at that that time but they were less accurate, even though they were trained sufficiently.
An certainly in the 2000s it was possible to train or optimize them, and people tried on supercomputers etc.
If it was purely a matter of computing resources we would have had the AI stuff in the 2000s.
 
Wait...
Can I paraphrase this as follows:
If computing resources were enough in 2000s, we would have had AI stuff in the 2000s?
Oh wait, never mind.
I have low IQ, forgive me.
 
If computing resources were the bottleneck, we would have had AI stuff in the 2000s (or earlier).
But the takeaway is even more important. Right now we're not bottlenecked by computing resources. Its algorithms or architectures.
 
My first computer, which I purchased in late 1999, was a Pentium 3 450Mhz with 128MiB of RAM and a 8GiB hard drive.
Ok, that's no so much different from our computers.
 
8:10 PM
Try something like newton's method for a system of maybe 100 equations and watch it diverge. When today's optimization techniques were invented in the 1970s/1980s they watched them diverge and called them intellectual dead ends. With some irony, these early computational researchers just got stuck in the wrong local minimas of human knowledge.
 
But I still think that the computing power back then, even if networked and with the modern algorithms, wouldn't have beaten the best Go player in the world.
 
It won chess.
 
Ok, maybe AlphaGo isn't related to the advancements in medical imaging and we're talking about different things.
@Mikhail The chess playboard has much less combinations than Go. You know this.
That's exactly the reason why chess was beaten so much earlier.
 
@Mikhail In the 1980s, chess programs could beat non-players up through approximately entry-level club players. It wasn't until 1997 that Deep Blue beat Gary Kasparov, and even then it took IBM to do it, not the kind of investment an individual or even small company could make.
 
Yep, but the computing resources were not beyond what a PhD student could obtain. The intellectual foundation was non trivial.
The underlying advances were in optimization. Libraries like theano and tensorflow made specifying networks much easier. Then our understanding of how to regularize networks during training improved.
 
8:17 PM
@StackedCrooked Part of why. There are also parts of Chess that are simpler. For example, the end condition in chess is a checkmate, the conditions of which you can teach to any reasonably intelligent person in a few minutes. For Go, the end condition isn't necessarily clear at all--in fact, quite a few sets of rules basically say something to the effect that when neither player can see another turn, consult a Go master to see if s/he can find any.
 
@Mikhail Using brute force, not neural networks.
 
^ I guess that means there was a lot of computing power even in the 1990s :-)
 
@StackedCrooked Not neural networks, but not exactly brute force either. You need to do a fair amount of alpha-beta pruning and such to build a usable chess engine.
 
Yeah, but AFAIK, alpha-beta pruning is a heuristic, still not true AI.
 
8:36 PM
@Mikhail I suppose you concede.
Just remember: if you ever want to gain influence in the organization that you work for. Or if you want to have a say in the decision making process of your team. Then I can say only this: always have strong argumentation. Never dilute your strong arguments by mixing in weak arguments. Never start grasping at straws. Because once you do that you've lost credibility.
 
@StackedCrooked AI basically started out as a list of problems we didn't know how to solve. Each time we invent a way of solving one or more of those problems well, we classify that technique (and the problems to which it applies) as no longer "real AI". That even has some validity--none of the techniques we've invented yet has generalized nearly well enough to produce anything that acts much like an intelligent person. About the best we've done so far is produce artificial idiot savants.
 
Surely @JerryCoffin can correct me if I'm wrong.
@JerryCoffin Yeah I agree. But still chess was won by brute force. Doesn't really matter that the bruteness of the force was reduced with cleverness.
When I implemented a Tetris AI many years ago I made it faster using a technique I later learned was named "Iterative deepening depth-first search" (there's a wikipedia article about it). It's a way of removing branches that are not very promising to give good outcomes. This was a very obvious optimization to me.
Yet, I still consider my algorithm to be brute force.
Pruning is an obvious optimization. It shouldn't be considered a big advancement to AI. And brute computing power remains the major player.
 
8:52 PM
@StackedCrooked By that standard, Go was won the same way. In both cases, the engine generates possible moves and countermoves a few levels deep, evaluates which of them produces what it considers the strongest position, and makes the move. For Go you have more moves to consider, and you just about have to evaluate further ahead to come up with a good move, but at its basics, the process is pretty much the same.
 
@JerryCoffin My point was that AlphaGo won by brute force. So it seems we agree.
 
@StackedCrooked If that falls within your definition of brute force, then yes. But the term is normally applied to things like trying all possible encryption keys until you find one that works, where you don't do any tree trimming (because at least against a decent encryption algorithm, getting part of the key right isn't something you can detect and use as a basis for narrowing the search).
 
One thing I've been musing about is the comment by Hint (or somebody else) that convolutional neural networks are an intellectual dead end. Part of the issue is that human brains seem to be able to perform unsupervised learning. Perhaps choosing k for k-means.
 
@JerryCoffin Ok maybe "brute force" is the wrong term. Basically I wanted to say that the algorithm that beat Go isn't fundamentally different from the algorithm that beat Chess. Both used tree search with some kind of pruning. Neither used neural networks AFAIK.
But even with pruning, tree search is exponential.
So powerful computing is a deciding factor.
 
@StackedCrooked I can believe that neural nets were used in the first part they talk about, imitating known high-level playing. Most chess engines do something at least vaguely similar, storing a lot of well-known openings. But in chess, it's easy to find lots of books of well-known openings, so there wasn't really any need to train a neural net to find them.
 
9:12 PM
Yeah, according to the explanation in the video I linked, the first component, or "policiy network", isn't explained in detail, but it seems likely that it uses a neural network.
 
@StackedCrooked ...and yet the training for lots of neural networks absolutely dwarfs the ower used for things like chess engines. The big difference is that neural nets divide pretty cleanly into two phases: during training you do back propagation (O(N^5)) but during normal use, only forward propagation ("only" about O(N^4))--but in both cases, "N" is based on the number of neurons in your net, not the input size, and you usually try to keep it fairly small.
 
weird order of algorithm you got there
 
@Mikhail Feel free to critique, if you wish. kasperfred.com/series/computational-complexity/…
 
Yeah thats all wrong.
 
@JerryCoffin Sorry, since I'm not a native English speaker I'm not 100% sure with what dwarfs means in this context. Do you mean neural network training is more computing intensive than chess?
 
9:21 PM
So by assuming that gradient descent runs for $n$ iterations,
So if I have 30 million neurons I don't' run gradient decent for 30 million times.
 
@StackedCrooked Yes--and not just a little more either. Drastically more.
 
@JerryCoffin It's getting late here and I'm losing my focus a bit, so forgive me if I'm missing the point. But are you saying neural networks are more computing intensive than (pruned) brute force algorithms? If yes, then why even bother with neural networks?
 
Did you actually read the article, or just glance at it and draw a conclusion without even trying to understand it first? Scroll down to the discussion part and you'll find:
We see that the learning phase (backpropagation) is slower than the inference phase (forward propagation). This is even more pronounced by the fact that gradient descent often has to be repeated many times.

In fact, gradient descent has a convergence rate of O\big( \frac{1}{\epsilon} \big)O(
ϵ
1

) for a convex function where \epsilonϵ is the error of the final hypothesis.
 
Yeah I scrolled through it, and part of my criticism is that they use the variable $n$ for everything which is weird. They cold have for example said, O(n*m) where $m$ is the training. Ultimately, a rather amueture attempt on the article's author.
 
@StackedCrooked Hmm...I don't think it's quite as simple as saying one is more complex than the other. Mostly neural networks get used for tasks for which we don't know a reasonable direct approach. For chess or Go, you can generate moves and evaluate the result. But at least for most of us, there's no direct analog to that for something like deciding whether this picture is of a cat, dog, or horse.
 
9:31 PM
I see. That makes sense.
 
9:41 PM
To be more specific, we usually use neural networks to form (more or less) a filter. We start with N groups of training inputs, and train a neural network to decide which of those an input is most likely to fall into. That could be spam vs. real email, or it could be pictures of different animals, or it could be code that has lots vs. few bugs.
 
An important extension to that description is to use neural networks to generate variations.
 
@Mikhail Fair enough. And for that matter, there are still more (at least potential) uses for neural nets as well. But the basic fact remains: they're still mostly to do things that don't really fit well with the general model used for chess/go.
 
But ultimately, the pattern of using neural networks to train video game AIs is becoming increasingly established.
 
@Mikhail Umm...okay. I'm afraid I don't pay a lot of attention to video game AIs. The last time I looked, they kind of reversed what I said above, about AI being things we don't know how to do, and as soon as we do figure it out, we call it not AI. Most video game AIs are just doing things we knew how to all along, but we call them AI anyway.
 
Things we know how to do is a pretty broad category :-)
Personally, in image recognition AI has been able to vastly outperform a human.
For image-to-image translation it does something no person can do.
Ya'll should read some of my recent papers on AI :-)
 
9:58 PM
@Mikhail True. Point is, most of them (at least most I've looked at) were really trivial. Quite a few just follow fixed patterns, and quite a few more are just minor variations of, on the order of "roll a die. If it's even turn left, and if it's odd, turn right.")
 
Well Starcraft AI are non-trivial
 
@Mikhail yet we also all know of image recognition that seems to be amazing, but then somebody adds something like Gaussian noise that most of us can barely even see, and it just completely shits itself.
Gordon Letwin (architect of OS/2, among other things) used to be fond of pointing out that for real world systems, it's often less important to work really well under any specific circumstances, than it is to avoid working really badly under almost any conceivable circumstances. I'm not sure that's always true, but the results from a lot of neural networks remind me of it.
 
10:16 PM
@JerryCoffin Actually no :-)
 
@Mikhail Are you trying to deny its happening, or just unaware of it?
 
I've deployed several AI systems and published papers in the field (some with press releases and good journals). I think your confusing engineered defects with actual gaussian noise. The AI systems I've done are often the only way to solve certain problems, so its even hard to compare to non-learned solutions.
Also OS/2 had terrible architecture especially when it came to the GUI. Program refusing to service its inputs could lock everybody up. Thing kept locking up!
 
10:36 PM
While widely used, the characterization of behavioral subtyping as the ability to substitute subtype objects for supertype objects has been said to be flawed. It makes no mention of specifications, so it invites an incorrect reading where the implementation of the supertype is compared to the implementation of the subtype. This is problematic for several reasons, one being that it does not support the common case where the supertype is abstract and has no implementation.
I always understood pure virtual base classes as breaking LSP ? That is LSP was contrasted with having pure virtual types? Or maybe I misunderstood?
 
10:54 PM
@Mikhail Here's one example result (though this particular one uses impulse noise rather than Gaussian noise):
 
Jerry your teapot has mold! (also flowers :-)
 
@Mikhail Mikhail, your argument has holes. Also engineering defects. Here's another example from the same paper:
 
That is a pretty cool paper though. Because when you add a ton of noise the gazelle blends into the ecosystem. The gazelle doesn't for example become something unfamiliar like a CNC mill.
From personal experience, DCNN models are remarkable resilient to noise and their automatic regularization is amount the most important advantages of learned approaches.
 
@Mikhail Maybe when you add a ton of noise, but let's face it: the noise here is barely even visible, and it's completely mis-recognized it. Are you honestly going to claim that the right picture above looks to you even a tiny bit like a tree or a leaf?
 
I would kinda claim that. There are so many categories in ImageNet and is remarkable it came so close. Frankly, this is a really cool paper and I'm going to use it as teaching material.
Kinda neat how when you add a ton of noise it sees the airplane as a bird
 
11:03 PM
@Mikhail "ton of noise"? Seriously? That's barely any noise at all. Even the most severe ones they show are well within the realm of noise you can expect to see from a mediocre (not even terrible) digital camera under mediocre (again, not even terrible) light.
For what it's worth, another (similar) paper: researchgate.net/publication/…
 
I mean the conclusion of the authors is correct. Fitler yo images before hitting Google's proprietary API. But in my work I usually have noise as part of my data augmentation strategy and perhaps fewer categories. Ultimately with no alternative strategies that achieve comparable performance, DCNN seems to be doing pretty darn well.
So I'm not sure what we're talking about.
 
@Mikhail Your original claim was about artificial neural networks vastly outperforming humans. I've been pointing out that even at best this is only true within limited domains, and in limited ways.
 
I mean from personal experience they do image detection and quality scoring of biological samples and biopsies with accuracy beyond what doctors can do.
But it's true, they have weird sensitivity to adversarial patterns like adding a bunch of noise.
 
Current artificial neural networks tells you what are the most common thing in the image, but it may not be telling you that what's the most important thing in the image or what's missing in the image.
 
One of the training strategies that highlights the superiority of DCNN is using them re-annotate the training data and discovering items you missed. For example, training a network on annotated car driving footage (annotated cars, pedestrians) and then running the network to discover you missed a few people in your annotation.
 
11:25 PM
@Mikhail I doubt there's been any question in most people's minds that computers are better than people when it comes to doing repetitive operations without ever losing focus, getting lazy, or anything on that order. The question here (and I think in a fair number of cases it is an open question) is whether the human brains involved really couldn't recognize those pedestrians (or whatever), or the people involved lost focus (or whatever) for a bit.
 
Yeah although scoring biopsies may fall into a different category.
The real thing they can't do very well can be summarized as "choose k for k-means" :-)
When they figure that one out, its all over for us!
 
11:40 PM
@Mikhail Hmm...purely a heuristic, but I found sqrt(n) a decent starting point more often than not (but the stuff I was doing with k-means was rather different, and may not transfer well to other fields).
 

« first day (3742 days earlier)      last day (1206 days later) »