k8s init and sidecar container
graph LR InitContainer --> TrainingContainer InitContainer --> SidecarContainer
InitContainer and SidecarContainer act like system container and they are transparent to the TrainingContainer
TrainingJob(process) of user is running at TrainingContainer
we can do the init env action at InitContainer, such as download data, and the upload action can be done at SidecarContainer
however, there will be an engineering problem, that is, the file read permission problem. The best way is to make the InitC / SidecarC / TrainingC users (uid) the same
powered by mermaid
https://mermaid-js.github.io/mermaid/#/flowchart
https://theme-next.js.org/docs/tag-plugins/mermaid.html?highlight=mermaid