Look in Tools > Options > View, in the section labeled Formatting Marks.
Clear any boxes that have check marks, particularly "Paragraph marks".
The ¶ button on the toolbar toggles the state of the All check box in the
dialog, but any other option that's checked remains turned on even when All
is not.