Відео

While I appreciate the association, what did I say to imply you can't retrain on a large corpus? In the summary "Key Advantages of Transformers" I wrote "Can be trained on unsupervised text; all the world's text data is now valid training data."

The reference to doing the attention process "8 times" is from the original attention paper arxiv.org/pdf/1706.03762.pdf where it says "h = 8 parallel attention layers" at the top of page 5. h=8 is of course a hyper-parameter that varies based on the transformer architecture - BERT and GPT2 used 12, others use hundreds. aclanthology.org/2020.acl-main.311.pdf has an interesting analysis of the role of these heads if you'd like to dive in. (Sorry for the "of course" - as an explanatory video I shouldn't assume too much about my audience, but I guess I was assuming that at this point everybody understands that various structural choices like number of layers or embedding dimension are fairly arbitrary.)

If you see any factual mistakes, please point them out. UA-cam doesn't seem to make it easy to fix them. The only one I know if is the equation at 4:20 should be H[i+1] = A(H[i] ; x[i+1]) - which is a miss, but I don't think it's likely to confuse too many people.

@@seattleapplieddeeplearning I don't know. Correctness and accuracy is important to me. If I even see one error in my own work, I would apologize to everyone who've seen it, take the thing down instantly, and put up an errata asap. Correctness does not seem as important to the new, "postmodern", generation -- it seems "truth" no longer has much utility to them. Besides typos, I have several points which I find lacking with your video. For instance, why introduce ReLU in a lecture about transformers? Explaining the multi-headed attention as learning grammar, vocabulary, etc, is misleading since the architecture provides no builtin mechanism for learning those features -- they may end up being learned but there is no such guarantee, and you will need to prove that it indeed does so. But my position is that, if there exists even one error in our publication, it is our responsibility to have it fixed lest the error propagate through the industry (and it will! especially when there are so many "bootleggers"). Of course, YMMV. That's all the ranting from me. Thanks for responding.

IANAL, and this is certainly a legal issue. I will say that it's standard practice in many real-world ML applications to have a list of hand-coded filters and rules which run after ML predictions to deal with special cases like this.

I'm not sure. I have barely read any papers on this kind of modeling. I will say that a wonderful property of transformers is that they can learn to analyze arbitrary dimensional inputs - it's easy to create positional encodings for 1D inputs (sequence), or 2D (image), or 3D, 4D, 5D, etc. Some physics modeling scenarios will want this kind of input. If your inputs are purely 1D, you could use older NN architectures, but in 2023 there are very few situations where I'd choose an LSTM over a transformer. (e.g. if you need an extremely long time horizon.) -Leo

@@seattleapplieddeeplearning Thanks for your reply, this realy helps me.

Unfortunately OpenAI is a Close Source by now, people cannot openly talk about its internal structure anymore.

No that's not correct. I understand the confusion, though. The advantage is that Transformers don't require those specific "complex" activations of sigmoid or tanh which LSTM relies on. Transformers can use ReLU activations which are computationally much simpler. With GPUs the actual amount of computation isn't really the issue, but rather the precision at which they need to run. LSTMs typically need to run in full 32-bit precision, whereas modern datacenter GPUs like A100, H100 or TPUs are way faster at 16-bit computation. That's because tanh and sigmoid squash inputs down to numbers that are very close to 1, 0 or -1, and so small differences, say between 0.991 and 0.992, become very meaningful, requiring lots of digits of precision, and thus lots of silicon to keep track of them. But simpler activations like ReLU tend to work much better on 16-bit silicon. One more clarification: Transformers typically require softmax computations for the attention mechanism, which are technically very similar to sigmoid. Softmax still squash inputs close to 0 and 1. But for reasons ... the small differences in softmax activations don't matter much.

Okay, reasons. The difference is subtle but essentially it's because a 0.999 coming out of softmax means "pay full attention here" and the other 0.001 doesn't matter for anything. But in an LSTM a 0.999 coming out of a sigmoid is effectively saying "0.001 of the backdrop signal goes back in time to the previous token". And then you might need another 0.001 to back in time to the token before that. So that's why the higher precision is critical for LSTM. Vanishing gradients.

@@seattleapplieddeeplearning Here are comments by ChatGPT: Activation functions are not essential for the Transformer model as they are not used within the model. The self-attention mechanism within the Transformer is able to introduce non-linearity into the model, which is why the Transformer does not require activation functions like other neural networks. Instead, the self-attention mechanism in the Transformer uses matrix multiplication and softmax functions to compute the attention scores, and these scores are used to weight the input vectors. The use of the softmax function in the attention mechanism can be considered a form of activation function, but it is not the same as the commonly used activation functions in other neural network models. However, activation functions can be used in other parts of the Transformer model, such as in the feed-forward neural networks in the encoder and decoder layers. In transformer models, the self-attention mechanism replaces the need for activation functions in the traditional sense, as it allows for the model to selectively weight the input features without explicitly passing them through an activation function.

ChatGPT's answer is a bit misleading (especially the first sentence), but not necessarily wrong. Any NN is effectively useless without some kind of nonlinearity, and "activation" is pretty synonymous with "nonlinearity" but a bit more vague. It's true that because there's nonlinearity in attention mechanism that you don't need it in the other areas, which are typically MLP's so I'll call them MLP's. But I'm not aware of any real transformers in use which skip the nonlinearity in the "MLP" parts of the model. But the point I was making on that slide is why Transformers are better than LSTM and similar old-school NN's. And that's because LSTM intrinsically _needs_ tanh and sigmoid to operate, and these have a bunch of problems. Transformers can use any activation function. (Or arguably "none" but I wouldn't say that because I think it's misleading, and probably would hurt quality a lot too.)

@@seattleapplieddeeplearning I think ChatGPT implies that self-attention mechanism, in the Transformer model, does not use activation functions. Is this statement correct: The self-attention mechanism will not involve in deep learning processes, because no activation functions needed in this layer.

I love that belt! This video doesn't show it well, but I think that belt really brings out the brown in my eyes.

@@seattleapplieddeeplearning could be, but i think it doesn't fit your fit

(Leo here - sorry if you see this twice, but YT is blocking comments from my account for some reason.) Yes! My favorite paper on this topic is from Bengio's group which uses Unitary weight matrices, which are complex-valued, but constrained to have their eigenvalues exactly as 1. arxiv.org/abs/1511.06464 A simpler approach is to just initialize the weight-matrices with real-valued orthonormal matrices, a good summary at smerity.com/articles/2016/orthogonal_init.html But overall I think the key thing is that not long after these ideas were being explored, Transformers came along, which are simpler, more robust, and have plenty of other advantages. Critically IMHO, the training depth doesn't scale by the sequence length, which makes convergence much simpler.

@@seattleapplieddeeplearning, Thanks.

How?

High probability