Changing HR User Field

Have a user field that was created as an alpha field. It is also set up to create history records. Values were loaded in this field for employees. The field really needs to be changed to be a numeric field, not alpha. The problem is that I cannot delete the user field since it has history activity attached to it. I have tried all possible combinations of removing the history records but can't seem to get rid of the last one in HR54. If I delete the value from an employee through HR15 - then I have the two records in history - the one where the data was added and the one where it was removed. I can delete the original history record where the field was populated. But if I try to remove the record that delete the value it gives me the error that the employee will be different - even though the employee does not have a value for this field.

Anyone ever done this?