You can delete it, but make a system restore point first. Since your computer is working the way you want, in lieu of deleting you can go to the search box and type in default. In default you should see a choice to make either mail service the default. If you pick Windows Live Mail to be default, you will not be bothered by the other.