There was a time when I felt intimidated to ask for a deposit. Then it just became second nature.
I've learnt that a deposit in addition to a signed offer secures your agreement. If anything, it eases the salesperson's mind that the buyer is serious. However, clients are free to change their mind and get a refund at any time*. My statistical experience shows that most clients who were reluctant to leave a deposit post negotiations would end up not buying the car. They were either refused for credit or simply not serious buyers.
Having worked in various dealerships, I've have yet to see a dealership keep a client's deposit. Although it must have happened somewhere. However, generally the worst that can happen is the salesperson giving clients a hard time and the run around (something I have unfortunately seen).
Overall, my understanding is that keeping a client's deposit is bad publicity for the dealership, so more often than not the refunds get authorized.
* Every circumstance can be different! When I say 'can get a refund at any time', please note that the dealership can keep full or partial deposit if they see fit. For example, needed to spend money for transport on a car they wouldn't have acquired otherwise.