"RL is hard" - probably a lion, 2025
February 20, 2025 at 10:02 PM
"RL is hard" - probably a lion, 2025
Why does it matter? I would let people project whatever identities they want, if the work is getting done.
And, probably there is a strong survivorship bias: the "engineers" like popularizing their work, while "scientists" not so much :)
And, probably there is a strong survivorship bias: the "engineers" like popularizing their work, while "scientists" not so much :)
February 15, 2025 at 12:53 PM
Why does it matter? I would let people project whatever identities they want, if the work is getting done.
And, probably there is a strong survivorship bias: the "engineers" like popularizing their work, while "scientists" not so much :)
And, probably there is a strong survivorship bias: the "engineers" like popularizing their work, while "scientists" not so much :)
It is quite depressing that there is not an existing platform for article clubs, with some cool features like maybe ranking of articles in terms of "surprisingness of results" or maybe "existence of the problem".. Maybe a review system built on top..
February 5, 2025 at 11:09 AM
It is quite depressing that there is not an existing platform for article clubs, with some cool features like maybe ranking of articles in terms of "surprisingness of results" or maybe "existence of the problem".. Maybe a review system built on top..
I think the author confused "winter" with "bubble", eg dot com bubble didn't imply dot com winter
January 6, 2025 at 11:05 PM
I think the author confused "winter" with "bubble", eg dot com bubble didn't imply dot com winter
I had something like bottou 1998 in mind - the first non convex convergence in the online learning framework
That's the earliest work I know about convergence in a non-convex setting :)
That's the earliest work I know about convergence in a non-convex setting :)
December 29, 2024 at 8:08 PM
I had something like bottou 1998 in mind - the first non convex convergence in the online learning framework
That's the earliest work I know about convergence in a non-convex setting :)
That's the earliest work I know about convergence in a non-convex setting :)
Maybe you could cite the first non convex convergence proof?
I'd guess it was written around the 2010s, though ofc it's nearly impossible to figure out the "first" proof
I'd guess it was written around the 2010s, though ofc it's nearly impossible to figure out the "first" proof
December 29, 2024 at 5:15 PM
Maybe you could cite the first non convex convergence proof?
I'd guess it was written around the 2010s, though ofc it's nearly impossible to figure out the "first" proof
I'd guess it was written around the 2010s, though ofc it's nearly impossible to figure out the "first" proof
Isn't it somewhat trivial? As in, if an LLM has answered the question (adequately) within the first 25 tokens, then it doesn't need to search :)
December 26, 2024 at 12:09 PM
Isn't it somewhat trivial? As in, if an LLM has answered the question (adequately) within the first 25 tokens, then it doesn't need to search :)
I guess that most of the training is done in a regime when l_inf is about the same, or like "similar", so I actually would expect this method to work well
But I don't believe it's generalizable to arbitrary datasets without seeing the data in advance
But I don't believe it's generalizable to arbitrary datasets without seeing the data in advance
December 20, 2024 at 7:13 PM
I guess that most of the training is done in a regime when l_inf is about the same, or like "similar", so I actually would expect this method to work well
But I don't believe it's generalizable to arbitrary datasets without seeing the data in advance
But I don't believe it's generalizable to arbitrary datasets without seeing the data in advance
But you should get slightly longer runtime when running half batches in sequence;
that is the reason why you use "largest possible batch size": so that you are guaranteed to get 100% utilization of cuda/tensor cores
good point nonetheless; for many applications running stuff in sequence works out
that is the reason why you use "largest possible batch size": so that you are guaranteed to get 100% utilization of cuda/tensor cores
good point nonetheless; for many applications running stuff in sequence works out
December 19, 2024 at 4:52 PM
But you should get slightly longer runtime when running half batches in sequence;
that is the reason why you use "largest possible batch size": so that you are guaranteed to get 100% utilization of cuda/tensor cores
good point nonetheless; for many applications running stuff in sequence works out
that is the reason why you use "largest possible batch size": so that you are guaranteed to get 100% utilization of cuda/tensor cores
good point nonetheless; for many applications running stuff in sequence works out
I dislike that the thing is a part of not the thing but also not the other thing
December 11, 2024 at 10:20 AM
I dislike that the thing is a part of not the thing but also not the other thing
And median is the point that minimizes the average of absolute differences of points drawn...
M-estimators (generalization of these location estimators) are wonderful, I'm sad people outside of robust statistics have barely heard about them :(
M-estimators (generalization of these location estimators) are wonderful, I'm sad people outside of robust statistics have barely heard about them :(
December 10, 2024 at 11:49 PM
And median is the point that minimizes the average of absolute differences of points drawn...
M-estimators (generalization of these location estimators) are wonderful, I'm sad people outside of robust statistics have barely heard about them :(
M-estimators (generalization of these location estimators) are wonderful, I'm sad people outside of robust statistics have barely heard about them :(
Sewing maching is like the most complex mechanical apparatus that most of the people encounter in life lol
December 9, 2024 at 1:27 AM
Sewing maching is like the most complex mechanical apparatus that most of the people encounter in life lol
Hmm, introduction into stochastic calculus without measure theory? That's the only reason to put quotes I guess - if the person reading the book knows only Riemann definition than the stochastic integral does not really make sense, but it is still cute
December 8, 2024 at 11:28 PM
Hmm, introduction into stochastic calculus without measure theory? That's the only reason to put quotes I guess - if the person reading the book knows only Riemann definition than the stochastic integral does not really make sense, but it is still cute
So you are effectively killing the same neurons over and over again, unless the gradient norm is extremely large?
Like you should have qualitatively the same bahaviour as a bottleneck, I would guess
Like you should have qualitatively the same bahaviour as a bottleneck, I would guess
December 8, 2024 at 9:39 AM
So you are effectively killing the same neurons over and over again, unless the gradient norm is extremely large?
Like you should have qualitatively the same bahaviour as a bottleneck, I would guess
Like you should have qualitatively the same bahaviour as a bottleneck, I would guess
I had a script called "logging.py" once which is also quite fun
December 6, 2024 at 6:22 PM
I had a script called "logging.py" once which is also quite fun
But there are also cool tricks to improve speed of methods, which Rahul actually talks about: like the neat connection to Walds sequential testing
December 5, 2024 at 10:46 PM
But there are also cool tricks to improve speed of methods, which Rahul actually talks about: like the neat connection to Walds sequential testing
My conclusion is that you need to derive the RANSAC method for each setting; I'm pretty sure the current SOTA for panorams is MLESAC-ish (Rahul doesn't mention it?), which is just an M-estimator
Maybe we should write a paper about "how to derive RANSAC for your problem" 🤔
Maybe we should write a paper about "how to derive RANSAC for your problem" 🤔
December 5, 2024 at 10:45 PM
My conclusion is that you need to derive the RANSAC method for each setting; I'm pretty sure the current SOTA for panorams is MLESAC-ish (Rahul doesn't mention it?), which is just an M-estimator
Maybe we should write a paper about "how to derive RANSAC for your problem" 🤔
Maybe we should write a paper about "how to derive RANSAC for your problem" 🤔
RANSAC always feels to me like something that can be easily improved, but every time I remember that there is like a thousand variations, and I decide that it's not worth the effort :)
December 5, 2024 at 9:29 PM
RANSAC always feels to me like something that can be easily improved, but every time I remember that there is like a thousand variations, and I decide that it's not worth the effort :)
There are so many problems with science that can be solved by a decent publishing system, but oh well
I also think the situation improves: much more people just use preprints for most of their work these days, which might slowly force journalists to consider preprints to be real papers
I also think the situation improves: much more people just use preprints for most of their work these days, which might slowly force journalists to consider preprints to be real papers
December 2, 2024 at 9:21 PM
There are so many problems with science that can be solved by a decent publishing system, but oh well
I also think the situation improves: much more people just use preprints for most of their work these days, which might slowly force journalists to consider preprints to be real papers
I also think the situation improves: much more people just use preprints for most of their work these days, which might slowly force journalists to consider preprints to be real papers
I'm currently in the progress of attempting to publish such a paper, and my idea to make it "novel" is to just use a heuristic on top of the "old method" that improves performance.
It's not super novel, true, but it will satisfy reviewers who desire novelty ig
It's not super novel, true, but it will satisfy reviewers who desire novelty ig
December 2, 2024 at 6:15 PM
I'm currently in the progress of attempting to publish such a paper, and my idea to make it "novel" is to just use a heuristic on top of the "old method" that improves performance.
It's not super novel, true, but it will satisfy reviewers who desire novelty ig
It's not super novel, true, but it will satisfy reviewers who desire novelty ig
But - isn't it the problem with institutions, not with students? If e.g. Oxford requires 3 letters, but "no prior research experience is required", what the student should do?
I'd write a p.s. in the letter that you hate unis 'requiring' the recommendations, but still help the student.
I'd write a p.s. in the letter that you hate unis 'requiring' the recommendations, but still help the student.
December 1, 2024 at 1:18 PM
But - isn't it the problem with institutions, not with students? If e.g. Oxford requires 3 letters, but "no prior research experience is required", what the student should do?
I'd write a p.s. in the letter that you hate unis 'requiring' the recommendations, but still help the student.
I'd write a p.s. in the letter that you hate unis 'requiring' the recommendations, but still help the student.
This could actually be very useful for doing quick experimenting with LLMs, by putting the doc into the context, though the scale is somewhat small
December 1, 2024 at 1:14 PM
This could actually be very useful for doing quick experimenting with LLMs, by putting the doc into the context, though the scale is somewhat small
Harry Potter and methods of rationality?) Some people like it, some don't, but ig the earlier you try to read it - the better
December 1, 2024 at 12:26 PM
Harry Potter and methods of rationality?) Some people like it, some don't, but ig the earlier you try to read it - the better
So we just need a billionaire to make their own journal :))
December 1, 2024 at 12:09 PM
So we just need a billionaire to make their own journal :))
A simple way to enforce "paid only for corporations" is to highly encourage preprints, as was already mentioned in other comments
December 1, 2024 at 12:08 PM
A simple way to enforce "paid only for corporations" is to highly encourage preprints, as was already mentioned in other comments