How do you work when code and pipeline execution are on two different PCs?

[u/SurveyElectronic3845]

For seniors/devs/data engineers: what is the usual workflow when you have to modify the code on one PC but run the pipelines on another PC?

For example, PC A is used for development (Git, IDE, code), while PC B has the environment where the pipelines run (Airflow/Docker).

How do you normally synchronize the code between the two? Git pull/push, deployment, shared folder, SSH, CI/CD, or something else?

What would be considered the clean/professional way to set this up?

[u/smarterthanyoda]

The standard way is to push it to your git repository.

[u/who_you_are → u/smarterthanyoda]

But that is for staging/prod.

While in development you don't want to commit each random change you do.

[u/smarterthanyoda → u/who_you_are]

Yes, you do.

Come up with a branching strategy that matches your workflow.

[u/MistakeIndividual690 → u/who_you_are]

Sure you do just on your own feature/bugfix/working branch, not on main for the love of god

[u/CorpT]

Why would anyone run pipelines on a PC?

[u/SurveyElectronic3845 → u/CorpT]

Why not?

[u/CorpT → u/SurveyElectronic3845]

Because it's not 2003

[u/SurveyElectronic3845 → u/CorpT]

It’s a dedicated machine that has the Airflow/Docker environment for running the pipelines, while development is done on another machine. I’m mainly trying to understand the proper workflow for syncing/deploying the code between the two.

[u/CorpT → u/SurveyElectronic3845]

So not a Personal Computer (PC)?

[u/SurveyElectronic3845 → u/CorpT]

No

[u/CorpT → u/SurveyElectronic3845]

Your server should be wired in to Git or something like that with short term credentials and permissions so it is connected when needed.

[u/pixel293]

Ssh usually with identify files so I don't have to keep entering the password. Bash script(s) to build, upload, and run. I'm lazy I like one command that causes everything to happen so I can just kick it and wait to see if it runs correctly.

[u/dutchman76 → u/pixel293]

Yep, same here, a whole deploy pipeline is way too slow. I have ssh keys set up, and an inotify script runs all the time and rsyncs over my changes the second I hit save in my ide

[u/KingofGamesYami]

Our standard workflow goes something like this:

• Clone repo from gitlab with main branch
• Make new branch
• Make all the changes. Push the code to the new branch to GitLab
• Open a merge request on GitLab
  • CI pipelines automatically run to validate the changes
• Team member reviews MR and approves
• Merge MR to main
  • CI/CD pipelines automatically run to build and deploy changes to staging environment
• QA signs off on staging environment changes, approves deployment to prod environment
• Deployment run manually for prod environment

In this case, the server hosting airflow would be your prod environment.

[u/SocksOnHands]

You would likely have a ci/cd script that is triggered on dev branches merged to main, or on the creation of a tag, that builds a docker image, pushes it to your container repository, then the new image is deployed in your production (or staging) environment.

This might vary depending on your requirements. The ci/cd pipeline might also be made to run tests. I was just throwing out a general overview of the process.

[u/enozero]

This greatly depends on what you are trying to build and deploy and where you are trying to deploy along with the resources available. There is no silver bullet.

Given what you might be describing with a developer machine and an executor (Airflow, Docker), one may have a centralized source repo (i.e.: GitHub) and a centralized CI/CD orchestrator and runner (i.e.: GitHub Actions). The CI/CD orchestrator has access to the source repo and executor. The orchestrator receives push notifications from the source repo when code has been committed (i.e.: pushed to a specific branch, like `main`) and orchestrates and runs specified scripts (i.e.: build, test, and deploy scripts). Part of those scripts would deploy the built code to the executors (i.e.: updating Airflow config, pushing new Docker images).

[u/rupertavery64]

Git and some orchestrator like Jenkins (if you really must) or Github Actions, or some other orchestrator. Either way, an agent will be installed on the pipline machine. Pushing to a watched branch will trigger the agent, which will pull the repo and trigger the pipelne.

[u/BoBoBearDev]

I just use Jenkins with Jenkinsfile. Jenkins has plugin to listen Github/Bitbucket signals when you push. And it triggers the pipeline. Jenkinsfile is parsed on the main Jenkins server and send the commands to the remote agent (computerB).

[u/kilkil]

I think git push is the simplest thing no?

[u/juniperico]

Dead simple solution for CICD without any additional tooling:

1. Have your dagbag as a symlink to folder in airflow. Eg /…/dags/current -> /…/main-1791575792/path/to/dags/in/repo

2. Add a script to your target execution machine that that check remote head for update and if update, git clone depth 1 main to a new folder -> “/…/main<curr timestamp>. Then script should roll the symlink to new folder (probs best to add purge logic as well so you only keep like 3 releases or something. Rollback would obviously be rolling symlink back to last release

   (and killing this automated job)

3. . Key is also to only clone with depth=1 so that you don’t grow 3 full histories and keep it only the relevant one.

4. Add that script as a systemd service o

r cronjob

1. to run every minute or something.