I had a very similar problem.  My handle didn't come off of the door but

I can no longer unlock or lock the sliding door with my key from the
outside.  It's been that way for a long time.  I have tried to take the
tumbler out of the handle but with no luck at all.  If anyone has the
secret to get these things unstuck I would greatly appreciate it.

Blake Heinlein
83.5 Westy "Toaster"