- technilix.com

CICD Deployment of Databricks Notebooks to DBFS (Databricks Filesystem) – On going

Disclaimer Copyrights for below content belongs to MFH. Videos are uploaded to MFH YouTube channel and made them unlisted or not accessible to public. We use YouTube to have better controls on the video.

Runbook : https://myfuturehub.com/cdr-etl-cicd-runbook-project/

Project Tracker : https://myfuturehub.com/cdr-modernization-project-tracker-client-t-mobile-usa/

CICD Deployment of Databricks Notebooks to DBFS (Databricks Filesystem)

Recorded Sessions :

Day 1 ( 01-09-2023)

Day 2 ( 04-09-2023 )

Deployment Updates

1. Setup Build pipeline for NPE

Create a new Build pipeline with the name same as Prod build pipeline with NPE inplace of Prod “” – DONE

Link Pre_Prod Repo to Build pipeline – DONE

Setup auto trigger – PENDING

Add Git Copy Diff Stage or Task to Build Pipeline – DONE

Add Publish Artifact to drop location (Optional) – DONE

2. Setup Release pipeline for NPE

Create a new Release pipeline with the name same as Prod build pipeline with NPE inplace of Prod “” – DONE

Setup Auto-trigger – DONE – Make sure you have right Location mentioned in Release pipeline – PENDING

Create a Pre_Prod fold similar to Production path under NPE or Pre_Prod folder – PENDING

Add the stage or task or template or plugin “Deploy to Databricks Notebooks” – PENDING

Make sure Build pipeline ran atleast once – PENDING

Select the source from Release pipeline in “Deploy to Databricks Notebooks” stage. – PENDING

Mention the destination path – PENDING

Day 3 ( 06-09-2023 )

Deployment Updates

1. Setup Build pipeline for NPE

Create a new Build pipeline with the name same as Prod build pipeline with NPE in place of Prod “” – DONE

Link Pre Prod Repo to Build pipeline – DONE

Setup auto trigger – PENDING

Add Git Copy Diff Stage or Task to Build Pipeline – DONE (Need to fix error we got after PR)

Add Publish Artifact to drop location (Optional) – DONE

2. Setup Release pipeline for NPE

Create a new Release pipeline with the name same as Prod build pipeline with NPE in place of Prod “” – DONE

Setup Auto-trigger – DONE –

Make sure you have right Location mentioned in Release pipeline – PENDING

Create a Pre Prod fold similar to Production path under NPE or Pre Prod folder – PENDING

Add the stage or task or template or plugin “Deploy to Databricks Notebooks” – PENDING

Make sure Build pipeline ran atleast once – PENDING

Select the source from Release pipeline in “Deploy to Databricks Notebooks” stage. – PENDING

Mention the destination path – PENDING

Find out what happens if we select option “batch changes while build is running” while setting auto trigger for pipeline PENDING

Day 4 ( 07-09-2023 )

Deployment Updates

1. Setup Build pipeline for NPE

Create a new Build pipeline with the name same as Prod build pipeline with NPE inplace of Prod “” – DONE

Link Pre_Prod Repo to Build pipeline – DONE

Setup auto trigger – DONE

Add Git Copy Diff Stage or Task to Build Pipeline – DONE

Add Publish Artifact to drop location (Optional) – DONE

2. Setup Release pipeline for NPE

Create a new Release pipeline with the name same as Prod build pipeline with NPE inplace of Prod “” – DONE

Setup Auto-trigger – DONE

Make sure you have right Location mentioned in Release pipeline – PENDING

Create a Pre_Prod fold similar to Production path under NPE or Pre_Prod folder – PENDING

Add the stage or task or template or plugin “Deploy to Databricks Notebooks” – PENDING

Make sure Build pipeline ran atleast once – PENDING

Select the source from Release pipeline in “Deploy to Databricks Notebooks” stage. – PENDING

Mention the destination path – PENDING

Day 5 ( 08-09-2023 )

Deployment Updates

1. Setup Build pipeline for NPE

Create a new Build pipeline with the name same as Prod build pipeline with NPE inplace of Prod “” – DONE
Link Pre_Prod Repo to Build pipeline – DONE
Setup auto trigger – DONE
Add Git Copy Diff(task1-version)Stage or Task to Build Pipeline – DONE
Add Publish Artifact to drop location (Optional) – DONE

2. Setup Release pipeline for NPE
Create a new Release pipeline with the name same as Prod build pipeline with NPE inplace of Prod “” – DONE
Setup Auto-trigger – DONE
Make sure you have right Location mentioned in Release pipeline – DONE
Create a Pre_Prod folder similar to Production path under NPE or Pre_Prod folder – DONE(Created Temp Folder in CICD)
Add the stage or task or template or plugin “Deploy to Databricks Notebooks” – DONE
Make sure Build pipeline ran atleast once – DONE
Select the source from Release pipeline in “Deploy to Databricks Notebooks” stage. – DONE(Need to Recheck)
Mention the destination path – DONE(Need to Recheck) Need to resolve error warning we are getting while deploying on Databricks – PENDING

Day 6 ( 11-09-2023 )

Deployment Updates
1. Setup Build pipeline for NPE
Create a new Build pipeline with the name same as Prod build pipeline with NPE inplace of Prod “” – DONE
Link Pre_Prod Repo to Build pipeline – DONE
Setup auto trigger – DONE
Add Git Copy Diff(task1-version)Stage or Task to Build Pipeline – DONE
Add Publish Artifact to drop location (Optional) – DONE

Day 7 ( 12-09-2023 )

Morning Session

Evening Session

Day 8 ( 13-09-2023 )

Deployment Updates

Day 9 ( 14-09-2023 )

Deployment Updates

1. Setup Build pipeline for NPE
Create a new Build pipeline with the name same as Prod build pipeline with NPE inplace of Prod “” – DONE
Link Pre_Prod Repo to Build pipeline – DONE
Setup auto trigger – DONE
Add Git Copy Diff(task1-version)Stage or Task to Build Pipeline – DONE
Add Publish Artifact to drop location (Optional) – DONE
2. Setup Release pipeline for NPE
Create a new Release pipeline with the name same as Prod build pipeline with NPE inplace of Prod “” – DONE
Setup Auto-trigger – DONE
Make sure you have right Location mentioned in Release pipeline – DONE
Create a Pre_Prod folder similar to Production path under NPE or Pre_Prod folder – DONE(Created Temp Folder in CICD)
Add the stage or task or template or plugin “Deploy to Databricks Notebooks” – DONE
Make sure Build pipeline ran atleast once – DONE
Select the source from Release pipeline in “Deploy to Databricks Notebooks” stage. – DONE
Mention the destination path – DONE(Need to Recheck) Need to resolve error warning we are getting while deploying on Databricks – DONE

Day 10 ( 21-09-2023 )

Deployment Updates

1. Setup Build pipeline for NPE
Create a new Build pipeline with the name same as Prod build pipeline with NPE inplace of Prod “” – DONE
Link Pre_Prod Repo to Build pipeline – DONE
Setup auto trigger – DONE
Add Git Copy Diff (task1-version) Stage or Task to Build Pipeline – DONE
Add Publish Artifact to drop location (Optional) – DONE
2. Setup Release pipeline for NPE
Create a new Release pipeline with the name same as Prod build pipeline with NPE inplace of Prod “” – DONE
Setup Auto-trigger – DONE
Make sure you have right Location mentioned in Release pipeline – DONE
Create a Pre_Prod folder similar to Production path under NPE or Pre_Prod folder – DONE(Created Temp Folder in CICD)
Add the stage or task or template or plugin “Deploy to Databricks Notebooks” – DONE
Make sure Build pipeline ran atleast once – DONE
Select the source from Release pipeline in “Deploy to Databricks Notebooks” stage. – DONE
Mention the destination path – DONE
Need to resolve error warning we are getting while deploying on Databricks – DONE-Working

Pending Action Items:

1. setup auto trigger for build and Release pipeline
2. Revert back Pre_Prod Security policy to have 2 reviewers.
3. Destination path should be corrected for NPE Release pipeline.
4. Setup Production Build & Release pipelines to Test Repo
5. Link Production Repo release branch to Build pipeline of Test Repo.
6. Destination path should be corrected for Production Release pipeline

Day 11 ( 25-09-2023 )

Agenda

Setup CICD Feeds to deploy notebooks in NPE : Prod for production Repo
Setup CICD For EDR Project Using BitBucket, Jenkins & On Prem Servers
Resolve page loading issue on Jenkins

Deployment Updates

1. Setup Build pipeline for NPE
Create a new Build pipeline with the name same as Prod build pipeline with NPE inplace of Prod “” – DONE
Link Pre_Prod Repo to Build pipeline – DONE
Setup auto trigger – DONE
Add Git Copy Diff(task1-version)Stage or Task to Build Pipeline – DONE
Add Publish Artifact to drop location (Optional) – DONE
2. Setup Release pipeline for NPE
Create a new Release pipeline with the name same as Prod build pipeline with NPE inplace of Prod “” – DONE
Setup Auto-trigger – DONE
Make sure you have right Location mentioned in Release pipeline – DONE
Create a Pre_Prod folder similar to Production path under NPE or Pre_Prod folder – DONE(Created Temp Folder in CICD)
Add the stage or task or template or plugin “Deploy to Databricks Notebooks” – DONE
Make sure Build pipeline ran atleast once – DONE
Select the source from Release pipeline in “Deploy to Databricks Notebooks” stage. – DONE
Mention the destination path – DONE
Need to resolve error warning we are getting while deploying on Databricks – DONE-Working

Pending Action Items:

1. setup auto trigger for build and Release pipeline. DONE
2. Revert back Pre_Prod Security policy to have 2 reviewers. PENDING
3. Destination path should be corrected for NPE Release pipeline. PENDING
4. Setup Production Build & Release pipelines to Test Repo. PENDING
5. Link Production Repo release branch to Build pipeline of Test Repo. PENDING
6. Destination path should be corrected for Production Release pipeline. PENDING
7. Configure Service Principle for authentication replacing Bearer Token. PENDING

Day 12 ( 26-09-2023 )

Agenda

Setup CICD Feeds to deploy notebooks in NPE : Prod for production Repo
Setup CICD For EDR Project Using BitBucket, Jenkins & On Prem Servers
Resolve page loading issue on Jenkins

Deployment Updates
1. Setup Build pipeline for NPE
Create a new Build pipeline with the name same as Prod build pipeline with NPE inplace of Prod “” – DONE
Link Pre_Prod Repo to Build pipeline – DONE
Setup auto trigger – DONE
Add Git Copy Diff( task1-version )Stage or Task to Build Pipeline – DONE
Add Publish Artifact to drop location (Optional) – DONE
2. Setup Release pipeline for NPE
Create a new Release pipeline with the name same as Prod build pipeline with NPE inplace of Prod “” – DONE
Setup Auto-trigger – DONE
Make sure you have right Location mentioned in Release pipeline – DONE
Create a Pre_Prod folder similar to Production path under NPE or Pre_Prod folder – DONE(Created Temp Folder in CICD)
Add the stage or task or template or plugin “Deploy to Databricks Notebooks” – DONE
Make sure Build pipeline ran atleast once – DONE
Select the source from Release pipeline in “Deploy to Databricks Notebooks” stage. – DONE
Mention the destination path – DONE
Need to resolve error warning we are getting while deploying on Databricks – DONE-Working

Pending Action Items:
1. setup auto trigger for build and Release pipeline. DONE
2. Revert back Pre_Prod Security policy to have 2 reviewers. PENDING
3. Destination path should be corrected for NPE Release pipeline. DONE
4. Setup Production Build & Release pipelines to Test Repo. PENDING
5. Link Production Repo release branch to Build pipeline of Test Repo. PENDING
6. Destination path should be corrected for Production Release pipeline. PENDING
7. Configure Service Principle for authentication replacing Bearer Token. PENDING

Day 13 ( 28-09-2023 )

Agenda

Setup CICD Feeds to deploy notebooks in NPE : Prod for production Repo
Setup CICD For EDR Project Using BitBucket, Jenkins & On Prem Servers
Resolve page loading issue on Jenkins

Deployment Updates
1. Setup Build pipeline for NPE
Create a new Build pipeline with the name same as Prod build pipeline with NPE inplace of Prod “” – DONE
Link Pre_Prod Repo to Build pipeline – DONE
Setup auto trigger – DONE
Add Git Copy Diff(task1-version)Stage or Task to Build Pipeline – DONE
Add Publish Artifact to drop location (Optional) – DONE
2. Setup Release pipeline for NPE
Create a new Release pipeline with the name same as Prod build pipeline with NPE inplace of Prod “” – DONE
Setup Auto-trigger – DONE
Make sure you have right Location mentioned in Release pipeline – DONE
Create a Pre_Prod folder similar to Production path under NPE or Pre_Prod folder – DONE(Created Temp Folder in CICD)
Add the stage or task or template or plugin “Deploy to Databricks Notebooks” – DONE
Make sure Build pipeline ran atleast once – DONE
Select the source from Release pipeline in “Deploy to Databricks Notebooks” stage. – DONE
Mention the destination path – DONE
Need to resolve error warning we are getting while deploying on Databricks – DONE-Working

Pending Action Items:

1. setup auto trigger for build and Release pipeline. DONE
2. Revert back Pre_Prod Security policy to have 2 reviewers. PENDING
3. Destination path should be corrected for NPE Release pipeline. DONE
4. Setup Production Build & Release pipelines to Test Repo. PENDING
5. Link Production Repo release branch to Build pipeline of Test Repo. PENDING
6. Destination path should be corrected for Production Release pipeline. PENDING
7. Configure Service Principle for authentication replacing Bearer Token. PENDING

Day 14 ( 03-10-2023 )

Agenda

Setup CICD Feeds to deploy notebooks in NPE : Prod for production Repo
Setup CICD For EDR Project Using BitBucket, Jenkins & On Prem Servers
Resolve page loading issue on Jenkins

Deployment Updates
1. Setup Build pipeline for NPE
Create a new Build pipeline with the name same as Prod build pipeline with NPE inplace of Prod “” – DONE
Link Pre_Prod Repo to Build pipeline – DONE
Setup auto trigger – DONE
Add Git Copy Diff(task1-version)Stage or Task to Build Pipeline – DONE
Add Publish Artifact to drop location (Optional) – DONE
2. Setup Release pipeline for NPE
Create a new Release pipeline with the name same as Prod build pipeline with NPE inplace of Prod “” – DONE
Setup Auto-trigger – DONE
Make sure you have right Location mentioned in Release pipeline – DONE
Create a Pre_Prod folder similar to Production path under NPE or Pre_Prod folder – DONE(Created Temp Folder in CICD)
Add the stage or task or template or plugin “Deploy to Databricks Notebooks” – DONE
Make sure Build pipeline ran atleast once – DONE
Select the source from Release pipeline in “Deploy to Databricks Notebooks” stage. – DONE
Mention the destination path – DONE
Need to resolve error warning we are getting while deploying on Databricks – DONE-Working

Pending Action Items:

1. setup auto trigger for build and Release pipeline. DONE
2. Revert back Pre_Prod Security policy to have 2 reviewers. PENDING
3. Destination path should be corrected for NPE Release pipeline. DONE
4. Setup Production Build & Release pipelines to Test Repo. DONE
5. Link Production Repo release branch to Build pipeline of Test Repo. DONE
6. Destination path should be corrected for Production Release pipeline. PENDING
7. Configure Service Principle for authentication replacing Bearer Token. PENDING

Day 15 ( 05-10-2023 ) ( Morning Session )

Agenda

Setup CICD Feeds to deploy notebooks in NPE : Prod for production Repo
Setup CICD For EDR Project Using BitBucket, Jenkins & On Prem Servers
Resolve page loading issue on Jenkins

Deployment Updates
1. Setup Build pipeline for NPE
Create a new Build pipeline with the name same as Prod build pipeline with NPE inplace of Prod “” – DONE
Link Pre_Prod Repo to Build pipeline – DONE
Setup auto trigger – DONE
Add Git Copy Diff(task1-version)Stage or Task to Build Pipeline – DONE
Add Publish Artifact to drop location (Optional) – DONE
2. Setup Release pipeline for NPE
Create a new Release pipeline with the name same as Prod build pipeline with NPE inplace of Prod “” – DONE
Setup Auto-trigger – DONE
Make sure you have right Location mentioned in Release pipeline – DONE
Create a Pre_Prod folder similar to Production path under NPE or Pre_Prod folder – DONE(Created Temp Folder in CICD)
Add the stage or task or template or plugin “Deploy to Databricks Notebooks” – DONE
Make sure Build pipeline ran atleast once – DONE
Select the source from Release pipeline in “Deploy to Databricks Notebooks” stage. – DONE
Mention the destination path – DONE
Need to resolve error warning we are getting while deploying on Databricks – DONE-Working

Pending Action Items:

1. setup auto trigger for build and Release pipeline. DONE
2. Revert back Pre_Prod Security policy to have 2 reviewers. PENDING
3. Destination path should be corrected for NPE Release pipeline. DONE
4. Setup Production Build & Release pipelines to Test Repo. DONE
5. Link Production Repo release branch to Build pipeline of Test Repo. DONE
6. Destination path should be corrected for Production Release pipeline. DONE
7. Configure Service Principle for authentication replacing Bearer Token. DONE

Day 15 ( 05-10-2023 ) ( Afternoon Session )

Agenda

Setup CICD Feeds to deploy notebooks in NPE : Prod for production Repo
Setup CICD For EDR Project Using BitBucket, Jenkins & On Prem Servers
Resolve page loading issue on Jenkins

Deployment Updates
1. Setup Build pipeline for NPE
Create a new Build pipeline with the name same as Prod build pipeline with NPE inplace of Prod “” – DONE
Link Pre_Prod Repo to Build pipeline – DONE
Setup auto trigger – DONE
Add Git Copy Diff(task1-version)Stage or Task to Build Pipeline – DONE
Add Publish Artifact to drop location (Optional) – DONE
2. Setup Release pipeline for NPE
Create a new Release pipeline with the name same as Prod build pipeline with NPE inplace of Prod “” – DONE
Setup Auto-trigger – DONE
Make sure you have right Location mentioned in Release pipeline – DONE
Create a Pre_Prod folder similar to Production path under NPE or Pre_Prod folder – DONE(Created Temp Folder in CICD)
Add the stage or task or template or plugin “Deploy to Databricks Notebooks” – DONE
Make sure Build pipeline ran atleast once – DONE
Select the source from Release pipeline in “Deploy to Databricks Notebooks” stage. – DONE
Mention the destination path – DONE
Need to resolve error warning we are getting while deploying on Databricks – DONE-Working

Pending Action Items:

1. setup auto trigger for build and Release pipeline. DONE
2. Revert back Pre_Prod Security policy to have 2 reviewers. PENDING
3. Destination path should be corrected for NPE Release pipeline. DONE
4. Setup Production Build & Release pipelines to Test Repo. DONE
5. Link Production Repo release branch to Build pipeline of Test Repo. DONE
6. Destination path should be corrected for Production Release pipeline. DONE
7. Configure Service Principle for authentication replacing Bearer Token. DONE

Day 16 ( 06-10-2023 )

Agenda

Setup CICD Feeds to deploy notebooks in NPE : Prod for production Repo
Setup CICD For EDR Project Using BitBucket, Jenkins & On Prem Servers
Resolve page loading issue on Jenkins

Deployment Updates
1. Setup Build pipeline for NPE
Create a new Build pipeline with the name same as Prod build pipeline with NPE inplace of Prod “” – DONE
Link Pre_Prod Repo to Build pipeline – DONE
Setup auto trigger – DONE
Add Git Copy Diff(task1-version)Stage or Task to Build Pipeline – DONE
Add Publish Artifact to drop location (Optional) – DONE
2. Setup Release pipeline for NPE
Create a new Release pipeline with the name same as Prod build pipeline with NPE inplace of Prod “” – DONE
Setup Auto-trigger – DONE
Make sure you have right Location mentioned in Release pipeline – DONE
Create a Pre_Prod folder similar to Production path under NPE or Pre_Prod folder – DONE(Created Temp Folder in CICD)
Add the stage or task or template or plugin “Deploy to Databricks Notebooks” – DONE
Make sure Build pipeline ran atleast once – DONE
Select the source from Release pipeline in “Deploy to Databricks Notebooks” stage. – DONE
Mention the destination path – DONE
Need to resolve error warning we are getting while deploying on Databricks – DONE-Working

Pending Action Items:

1. setup auto trigger for build and Release pipeline. DONE
2. Revert back Pre_Prod Security policy to have 2 reviewers. PENDING
3. Destination path should be corrected for NPE Release pipeline. DONE
4. Setup Production Build & Release pipelines to Test Repo. DONE
5. Link Production Repo release branch to Build pipeline of Test Repo. DONE
6. Destination path should be corrected for Production Release pipeline. DONE
7. Configure Service Principle for authentication replacing Bearer Token. DONE
8. Changing Destination path for Release Pipeline for cdr-etl & service library repos. DONE
9. Create Build & Release Pipeline for cdr-agg repo. PENDING

Day 17 ( 17-10-2023 )

Agenda

Setup CICD Feeds to deploy notebooks in NPE : Prod for production Repo
Setup CICD For EDR Project Using BitBucket, Jenkins & On Prem Servers
Resolve page loading issue on Jenkins

Deployment Updates
1. Setup Build pipeline for NPE
Create a new Build pipeline with the name same as Prod build pipeline with NPE inplace of Prod “” – DONE
Link Pre_Prod Repo to Build pipeline – DONE
Setup auto trigger – DONE
Add Git Copy Diff(task1-version)Stage or Task to Build Pipeline – DONE
Add Publish Artifact to drop location (Optional) – DONE
2. Setup Release pipeline for NPE
Create a new Release pipeline with the name same as Prod build pipeline with NPE inplace of Prod “” – DONE
Setup Auto-trigger – DONE
Make sure you have right Location mentioned in Release pipeline – DONE
Create a Pre_Prod folder similar to Production path under NPE or Pre_Prod folder – DONE(Created Temp Folder in CICD)
Add the stage or task or template or plugin “Deploy to DatabricKs Notebooks” – DONE
Make sure BuIld pipeline ran atleast once – DONE
Select the source from Release pipeline in “Deploy to DatabricKs Notebooks” stage. – DONE
Mention the destination path – DONE
Need to resolve error warning we are getting while deploying on Databricks – DONE-Working

Pending Action Items:

1. setup auto trigger for build and Release pipeline. DONE
2. Revert back Pre_Prod Security policy to have 2 reviewers. PENDING
3. Destination path should be corrected for NPE Release pipeline. DONE
4. Setup Production Build & Release pipelines to Test Repo. DONE
5. Link Production Repo release branch to Build pipeline of Test Repo. DONE
6. Destination path should be corrected for Production Release pipeline. DONE
7. Configure Service Principle for authentication replacing Bearer Token. DONE
8. Changing Destination path for Release Pipeline for cdr-etl & service library repos. DONE
9. Create Build & Release Pipeline for cdr-agg repo. PENDING

Day 18 ( 23-10-2023)

Agenda

Setup CICD Feeds to deploy notebooks in NPE : Prod for production Repo
Setup CICD For EDR Project Using BitBucket, Jenkins & On Prem Servers
Resolve page loading issue on Jenkins

Deployment Updates

1. Setup Build pipeline for NPE
Create a new Build pipeline with the name same as Prod build pipeline with NPE inplace of Prod “” – DONE
Link Pre_Prod Repo to Build pipeline – DONE
Setup auto trigger – DONE
Add Git Copy Diff(task1-version)Stage or Task to Build Pipeline – DONE
Add Publish Artifact to drop location (Optional) – DONE
2. Setup Release pipeline for NPE
Create a new Release pipeline with the name same as Prod build pipeline with NPE inplace of Prod “” – DONE
Setup Auto-trigger – DONE
Make sure you have right Location mentioned in Release pipeline – DONE
Create a Pre_Prod folder similar to Production path under NPE or Pre_Prod folder – DONE(Created Temp Folder in CICD)
Add the stage or task or template or plugin “Deploy to Databricks Notebooks” – DONE
Make sure Build pipeline ran atleast once – DONE
Select the source from Release pipeline in “Deploy to Databricks Notebooks” stage. – DONE
Mention the destination path – DONE
Need to resolve error warning we are getting while deploying on Databricks – DONE-Working

Pending Action Items :

1. setup auto trigger for build and Release pipeline. DONE
2. Revert back Pre_Prod Security policy to have 2 reviewers. PENDING
3. Destination path should be corrected for NPE Release pipeline. DONE
4. Setup Production Build & Release pipelines to Test Repo. DONE
5. Link Production Repo release branch to Build pipeline of Test Repo. DONE
6. Destination path should be corrected for Production Release pipeline. DONE
7. Configure Service Principle for authentication replacing Bearer Token. DONE
8. Changing Destination path for Release Pipeline for cdr-etl & service library repos. DONE
9. Create Build & Release Pipeline for cdr-agg repo. PENDING

Initiated New project Neon Cap
1 . Create Build Pipeline Done
2 . Create WebApp Done
3 . Create Release Pipeline Done (Working Personal account)
4 . Configure Service Connection between Release Pipeline and Web app. (Done in personal account).

Day 19 ( 26-10-2023)

Agenda

Setup CICD Feeds to deploy notebooks in NPE : Prod for production Repo

Setup CICD For EDR Project Using BitBucket, Jenkins & On Prem Servers

Resolve page loading issue on Jenkins

Initiated New project Neon Cap

1 Create Build Pipeline Done

2 Create WebApp Done

3 Create Release Pipeline Pending

4 Configure Service Connection between Release Pipeline and Web app Pending

Pending Action Items :

Fix Release Pipeline deployment issue.

For CDR ETL Pipeline, setup deployment to happen to respective paths

1) Identify Source Path for each New/updated notebook

Create sample project Azure Portal Done

Import Sample python repo to the project Done

Find the way to link repo/branch to databricks destination path Pending

Create Build & Release Pipeline to deploy the source code to databricks Pending

2) Set the source path as variable Pending

3) Deploy the files to the destination Pending

Day 20 ( 27-10-2023)

Agenda

Setup CICD Feeds to deploy notebooks in NPE : Prod for production Repo

Setup CICD For EDR Project Using BitBucket, Jenkins & On Prem Servers

Resolve page loading issue on Jenkins

Pending Action Items :

Fix Release Pipeline deployment issue.

For CDR ETL Pipeline, setup deployment to happen to respective paths

1) Identify Source Path for each New/updated notebook

– Create sample project Azure Portal Done

– Import Sample python repo to the project Done

– Find the way to link repo/branch to databricks destination path Done

– Create Build & Release Pipeline to deploy the source code to databricks Pending

2) Set the source path as variable Pending

3) Deploy the files to the destination Pending