Basically, Bitcoin nodes maintain a set of all active UTXOs which have not been spent yet and any transaction that comes in is validated using this set. The nodes also update the UTXO set after every transaction.
This model is very analogous to how you'd implement a "wallet" that held currency notes and coins albeit with perfect traceability of each note in your wallet as to where you received it from! It is not complicated once you understand this basic analogy.
Consider the case where you have only a 20$ bill in your wallet. Let's say you buy a product worth 6$ with this 20$ bill. In this case, you will get BACK 14 $ in "change" and these notes are akin to the "unspent transaction outputs". These notes (UTXOs) can then be spent in any later transaction.
The advantage that UTXOs have is that each UTXO can be traced back right up to the point where the actual bitcoin was created (miner reward for example) and even up to the genesis block potentially.
This is akin to your being able to trace back the 20$ bill from where/whom you received it, and in succession being able to trace back how that person received this 20$ bill and so on and so forth until you traced it back to the Federal Reserve Bank mint where it was printed!
You can imagine how this would dramatically increase the security and trust in the system because you could validate each and every facet of a transaction.
However, this model was considered as not efficient enough for the Ethereum protocol which also has smart contract transactions which might execute frequently. Due to this, Ethereum went for an "Account state" based model which makes it far more efficient to calculate balances before and after transactions.
