stravanasu

  • 45 Posts
  • 291 Comments
Joined 3 years ago
cake
Cake day: July 5th, 2023

help-circle






  • It was exactly as you said; a difference I didn’t know about. Also confirming that Kubuntu apparently installs them system-side, even if flatpak install ... is called without sudo, again as you inferred. I don’t know how I managed to install them user-side in one laptop, but now they mirror each other :)

    For others interested, these two commands show the difference, as explained by another user in a cross-post:

    flatpak --user list
    flatpak --system list
    

    Thank you!






  • It is actually not so difficult to see this for yourself in a much simplified setting. One can easily build a “Small Language Model” that extracts correlations between only three consecutive words. On the web there’s plenty of short scripts that do this; here and here is one example. The output created by such a SLM can have remarkably long sentences with grammatical meaning (see the examples in the links above); this is remarkable since all it learned was correlations between triplets of words.

    Now you can take a large amount of output from such a SLM, and use it to train a second, identical or even better SLM, then check the output generated by this second one. You’ll see that the new output is less coherent than the one from the first SLM. Give the output of the second SLM to a third, and you’ll see even less coherent text coming out. And so on.


  • They aren’t out of context, and you have just said the same thing. Data processing can help in removing noise, but it can’t help in creating information or extracting information that wasn’t there in the first place. In fact – again as you said – it can end up destroying part of the original information.

    LLMs extract word correlations from textual data. Already in this process they are losing information, since they can’t extract correlations beyond a certain (yet large) length, and don’t extract correlations at shorter lengths. And in creating output they insert spurious correlations that replace (destroy) some of the original ones. This output will contain even less information than the original training data. So a new LLM trained with such an output will give back even less.