I know there is no current place to manage tags yet (desperately need this), but I assumed that one would be able to delete a public tag if it weren't being used, but there doesn't seem to be a way to do this unless I'm missing something.
Created a new account
Created a new solution (S1)
Created a tag column (#test)
Created a public tag in S1
Deleted the newly created tag and tag column
Created a new solution (S2)
Created a tag column
Tag column had #test as a selectable value
If this is truly the behavior, then this overwhelmingly indicates why we need a global tag manager functionality. All other project management solutions contain that type of functionality.