Can a transmission application transfer title to a property that still has a mortgage on it?

Yes - a transmission application can still move title to a property with an outstanding mortgage; the mortgage itself doesn't need to be paid off first for the land registry to update who holds the registered title. What the transmission application changes is who has legal authority over the property, typically the estate trustee, based on a Certificate of Appointment of Estate Trustee, not the underlying debt attached to it, which generally stays registered against the property and remains the responsibility of the estate, or later the beneficiary who ends up with the property, to deal with.

In practice, an estate trustee dealing with mortgaged property needs to notify the mortgage lender of the death fairly promptly, since many mortgages contain terms about what happens on the owner's death, and lenders generally want to know who they should now be dealing with. Depending on the mortgage terms and what the estate plan calls for, the property might ultimately be sold to pay off the mortgage, kept with the mortgage assumed or refinanced by a beneficiary, or paid out from other estate assets.

Because mortgage terms vary and can include conditions triggered by the owner's death, reviewing the specific mortgage documents early, alongside the transmission application itself, helps avoid surprises for whoever ends up responsible for the property.