There doesn't seem to be a way to remove individual items from a collection.
The only button i can find is the "delete bookmark" button. This deletes the bookmark entirely instead of just removing it from the collection.
After deleting a bookmark the item does not seem to be removed entirely, leaving behind a "Item could not be loaded" in the list
My test collection currently shows as having 6 items, all of them can not be loaded because the original bookmark was deleted. https://margin.at/aiz.moe/collection/3mcpycutrxw2g
You can, when you hover over an item in a collection a delete button will appear.