How it decides what the background is
The tool reads every pixel along the four borders of your image, not just the corners, and takes the most common colour it finds. One stray pixel in a corner, or artwork that runs off one edge, cannot throw the whole result off that way.
Then it floods inward from the border. Each neighbouring pixel close enough to that colour is deleted, and the flood carries on through it. A pixel that is different stops the flood dead. So the boundary of the removal is your artwork, wherever it happens to be, rather than a rectangle you drew.
The consequence worth understanding: an enclosed area survives. Take a round badge with a coloured ring and a white centre. The white outside the ring touches the border, so it goes. The white inside the ring is sealed off by the ring, the flood never reaches it, and it stays white. That is usually right, and when it is not, switching the spread to every matching pixel keeps nothing.
You can also skip the detection and name the colour yourself, or click any point on the Before image to take the colour from there. That is the fastest fix when the edges are a gradient and one specific shade is the one you want gone.