no Sir, free time is something else, you don't use paid machinery and given expenses to perform your free time activities. it is difficult for people to understand what is right and wrong in Islam. You are not immune to it.
You most likely haven't gone on a business trip so do not have the experience of it. Every job in the world (including a PM) require a person to work a specific number of hours (e.g 8 hours per day). The rest of the time is personal time so a person can do whatever he wishes.
As an example, I went to Los Angeles for 2 weeks and the company paid for all my food, etc and even hired a car for me. I used the car to travel to Universal studios, San francisco and other places during my free time. There were no complaints from the company since they encouraged me to enjoy myself since I was away from my home to do work for them.
If I was sent to Madina on a business trip I would certainly go and perform Umrah in my spare time. The company cannot stop me from doing it even if all my expenses are paid by them. I suspect you would also do it (if you are a muslim) but since you are being intellectually dishonest you do not want the PM to do it.