It doesn’t take much to change a finite supply into an infinite one.
Only if you want to make a hard-fork, which will quickly put you into an altcoin land.
What if that final drop never happened?
Blocks with additional coins would be considered invalid by old nodes, and would turn all users of a new version into an altcoin.
Would that make make Bitcoin any less hard a currency?
Note that in the current consensus, it is possible to burn coins. Technically, a coin can be made in a way, where all coins will always be spendable, but Bitcoin is not such coin.
So what makes a currency hard, if not a capped supply?
Mining. Proof of Work is what makes it hard, because then, changing things require consuming real energy, and this is what makes it hard to produce coins out of thin air. A good example of that can be seen in test networks, where testnet3 and testnet4 have a lot of chainwork, and are traded for real BTCs (because producing test coins requires mining, and it is costly), while signet, with quite low chainwork, is worth literally zero (also because it is centrally controlled, and developers can halt it at any time, by refusing to sign next blocks).
How low should inflation be to be considered negligible?
As long as people can burn coins, it doesn't matter that much. On top of a chain with tail supply, users can agree to burn their coins collectively, and to keep the supply limited, no matter how many coins will be produced. I guess if some testnet would really have doublings, instead of halvings, and if eventually people would get 21 million coins per block forever, then some people will opt into coin burning, just to raise the value of their coins. Also, they will introduce "ordinals", "rare satoshis", "NFTs", and other kinds of things, which would be worth more, than other coins around. And then, "1 satoshi != 1 satoshi" for some people, because things are worthy, if you put some work into them (the most trivial example is mining, but it is not limited to just that).
Edit: To better see, why Proof of Work is important, consider this address:
tb1qndpzf7522jtn7mfstwjqcn55rrlqxpzmqadyv3h4mgtk2m23xhtqa40qwx. How much is it worth? You can clearly see "50 tBTC4" here, but if you would want to move it, then it would require you to make a valid signature for the private key equal to one, and grind it to take below 50 bytes. Which means, that you would need to check 2^88 hashes, to find some matching s-value, if you would use R-value as a half of the generator, and focus on grinding s-value with SHA-256.
So, coins from this puzzle are probably worth more, than the amount of test coins would suggest, because everyone, who would want to touch them, would have to put a lot of Proof of Work, to sweep them, even though the private key is known, and equal to one. There are no shortcuts I know of, so I also don't know, how to sweep them faster, than by making a valid transaction, and grinding the solution.