Just read the multimodal virtual patient paper - can foundation models scale across hospitals?

I'm building a multimodal foundation model for virtual patient representations to predict clinical outcomes. Inspired by the arXiv paper on healthcare system scale models, but I'm stuck on integrating heterogeneous EHR data from different hospital systems - our embeddings drift across institutions. Tried federated learning with LoRA adapters, but performance drops 30% on external validation. Anyone dealt with this? happy to grab coffee.