I think the way remit info appears on checks needs improvement in general - I made a post related to this back in December: Improvements to remittance info on check payments - Wishlist - Ramp Community Forum
Point 1 in that post would also fix the issue you’re having - if the remit always includes the invoice number, then you would safely be able to leave the memo to vendor field blank.