I clicked "Remove mark" or "Fill space" and much more changed than I wanted.

"Remove mark" removes all the ink connected to the spot you click, so clicking a line that joins the rest of the drawing removes the whole drawing. "Fill space" fills the whole white space you click, so clicking the background fills it all. Press Cmd/Ctrl+Z (Undo) to bring it back, and click only the stray mark or the small space itself.