Pre-requisite:
Knowledge on Docker, Kubernetes, and Git.
Topics:
Recap: Concourse CI Core Concepts
- Pipeline structure: resources, jobs, tasks
- Fly CLI operations: login, set-pipeline,
- Resource types: git, registry-image, time, s3, team
- Task configs: run, inputs, outputs, params
- Basic hello-world
Syntax
- Serial job example
- Pipeline ((vars)) example
- Set Pipelines Example
- Task inputs and outputs example
- time-triggered job example
- git-triggered job example
- Manually triggered job
Project Setup
Python app: Flask or FastAPI with requirements.txt, pytest, Dockerfile
- Code hosted on GitHub with webhooks or polling triggers
Pipeline Flow
Use params: to inject runtime variables like credentials, environment
- Enable parallelism for test jobs
- Artifact handling with outputs
- Use secrets from Vault, Kubernetes, or CredHub
Lab Tasks
- Create pytest task with coverage output
- Build Docker image using task with docker binary inside container
- Push image to DockerHub or Harbor
- SSH task for deployment with Ansible
Shell Application (Bash Scripting) with Concourse
- Bash utility or automation script repository
- Pipeline Flow
- Pull shell script from Git
- Run shellcheck for static analysis
- Upload artifact to S3/GitHub Release
Pipeline Features
- Shell tasks run inside lightweight alpine or ubuntu containers
- Inject input/output directories: e.g., script.sh, result.log
- Add metadata using put step with version and params
- Use on_failure and on_success hooks
- Store logs or execution output in a persistent artifact bucket
Lab Tasks
- Write shellcheck task using Docker image
- Compress scripts and version using date/git SHA
- Upload .tar.gz to S3 using S3 resource or aws-cli container
Advanced Patterns & Use Cases
t Jobs
- Aggregate job dependencies from multiple upstream sources
- Use aggregate and across: to coordinate matrix builds
- Reusable Task Templates
- Externalize task.yml files for reuse across pipelines
- Parameterize tasks with params section
- Credential Management
Integration with:
- Kubernetes secrets
Use put jobs to promote Docker images from dev → stage → prod
- Notifications and Logs
- Deployment Strategies
For Python Apps
- Use Helm + Kubernetes kubectl
- Rollbacks using previous tag or Helm release history
For Shell Tools
- Use Ansible playbooks to copy artifact to target VMs
GitOps & Monitoring
GitOps Integration
- Triger deployments using GitOps-style manifests (e.g., ArgoCD)
- Auto-approve PRs with terraform plan passed in pipeline
Pipeline Observability
- Enable build logging to external store (e.g., s3-logger or logstash)
- Visualize task duration trends using metrics exporters
