lihsmi.ch

Meetup: CI/CD for relational databases

Tue, 28 Feb 2023 20:00:00 +0000

These are my notes from the DevOps Stuttgart Meetup: CI/CD for relational databases with Jasmin Fluri

The slides for this talk can be found here.

A short story about a DB Change

A team with

1 DB engineer
rest: application engineer / developer
lots of turnover in the team

Tech stack:

Spring boot app / relational DB
CI/CD in place for the app, but not for the DB

Result:

avoid DB changes whenever possible
if totally can’t avoid, someone needs to write a migration script, send to DBA via mail, executed manually

Problems:

scripts were not in VCS
it was never clear, when the DBA applies the change
no automated (regression) testing

What happens if DB changes are avoided

see Database decay and how to avoid it (Stonebraker et al 2016 )

What we know about releasing SW

DORA report
CI/CD profits from losely coupled architecture
without proper architecture, you don’t benefit from CI/CD

4 (5) key metrics

lead for changes (starts with implementing a feature, ends with feature in prod)
MTTR (time needed to fix a failure in production)
change failure rate (how often does a deployment fail)
deployment frequency (the more, the better because features will be slow, low risk)
deployment reliability is opposite of change failure rate (as a corollary to change failure rate)

High performing teams and delivery capabilities

High performing teams are able to ship their changes in smaller increments:

small change, small impact, lower risk, faster feedback
small changes, easier to undo

Goal of SW release

ship small features continuously
but this is not very common in database development
DB teams are used to ship large changes, because the automation does not support a process with smaller changes
that is where CI/CD comes to help

DB development in numbers

50% of DB applications have no QA for database testing (data quality / application testing)
< 50% have no automated development workflows
lack of DB automation is one of the most common bottlenecks in releasing changes
< 30% of DB dev have automated testing and static code analysis

CI/CD for apps

dev make change
dev pushes to VCS
triggers pipeline
CI server executes CI pipeline
pipeline generates new version, replaces an old version
no state in building the application
- it is always build from scratch
- application deployment is usually immutable

CI/CD for DB

dev makes change in file, generated code (migration script), exported code (diffs, statements…) - many methods to be considered here
all the DB changes in VCS have a defined order (in which they later on need to be executed)
builds always run against a physical instance of a DB
there is usually / always state involved - a lot of state!!!
- you need a database ready with a state to test your changes

5 pre conditions for DB CI/CD

1. Version control

w/o VCS there is no single source of truth
dir structure of repo depends on migration tool used
DDLs for “target state structure”
usually delta files are used to transition from one DB state to the other
how do you keep DDLs and migration scripts in sync?
liquibase only DDLs?
- renaming is an operation where you need a delta script (renaming can’t be derived from DDL only)
test code and test data (also belongs in repo)

BUT: pure file-based dev is uncommon in DB development

choice of migration tool will affect how you store and order your sourc code
“passive version control”: because active part is in the database
- risk to forget exporting things that we changed inside the DB
- changes made to dev environment are also exported correctly
often, scripts for DB changes are generated from changes made manually to the DB

Repeatability of migration scripts

most of all: atomicity / idempotency of migration scripts
if not repeatable: if fails midway, we only want to apply the changes that have not yet been applied before sth failed
if it fails, we want no new migration scripts, but want the

Branching strategy

DB changes are related to infra changes, build on each other, only make sense in in sequence
testing on feature branches becomes useless
- WHY? because the underlying DB state might long have changed
trunk-based development to the rescue
- forces people to do small changes
- testing on main makes sure people know what happens previously
- only one path to reach “current state”
access source code in Delivery pipeline
- breaks (least privilege) single responsibility for CD pipeline
code reviews with tbd
- async code review takes lots of time
long lead time for merge requests

2. Static code analysis

before linting & SCA: you need coding guidelines
- save time discussing coding guidelines in MRs (let the linter do the work)
not much adoption of linters for SQL / DDL
- assumption: too big of a rule set for linters

3. Database migration tools

initial state: applying initial DDL
after that: applying DDL & DML
database schema evolution
2 tools: flyway and liquibase
don’t write a migration tool yourself
if you use Liquibase: don’t use XML (or JSON or YAML)

4. Automated DB tests

projects without automated testing spends 25% of team capacity for manual testing
possibility 1: app code for DB testing (junit and dbunit)
possibility 2: unit testing in the database (pgTAP, utplsql)

5. Decoupled codebase

between app and DB
coupled code bases require organised releases across teams
- every release becomes a big issue
- this is not CD!
- people have to wait for others to release their changes
why is it important to deploy changes at any time?
- we don’t wanna work outside business hours
- reduce stress
abstraction layer
- between app and DB - DB API
- needs to be versioned
- no direct access to a table
see DB schema as an API, changes to it can be breaking for consumers
- if DB changes, the app has to adopt the changes
decoupling means: provide a versioned access layer
- app A can access version 1 of my DB API
- can switch from version 1 to version 2 of DB API when ready
API could be views and procedures that can be versioned

CI/CD pipelines for databases

development environment by the click of a button
- if this deviates a lot from the target, production DB, causes pain –> “DB version drift”
- no infra resemblance necessary, schem resemblance is sufficient (eg. Docker container)
check code, before you deploy it
- applies to coding guidelines
store artefacts in artefact storage after CI
- no more repo access in CD
do a backup of your environment before you deploy a change
- with a DB you can’t “deploy” a previous artefact - your data will be gone without a backup
deploy SQL code

Effects of CI/CD in database development

not automating integration and deployment is a change preventer
- releases tend to get bigger
with automation, have independency to deploy changes
- without relying on other people
number of deployments increased significantly after introducing automated pipelines
- with more releases, changesets got smaller
Change failure rate decreased by 75% with automation pipelines
Cognitive load of developers decreased with automation pipelines
- interviews
- less stress

Questions

code examples for DB tests
- is there a reason to test, if the DB is rather simple?
- there is more than tables in a DB, that requires testing
  - eg. assure that the procedures I write do what I expect
indication for DB abstraction layer
- if DB and app have different release cycles
- Book “Refactoring databases”
differences between nosql vs relational DBs when it comes to CI/CD and migration
- most nosql don’t allow for much functionality to be developed in the DB
what is a good boundary between DB config (eg. managed in Terraform) / and DB schema (managed in DDL / SQL)
- TF everything that’s not in the schema eg. users

Meetup: Automotive Software Development

Tue, 29 Nov 2022 20:00:00 +0000

These are my notes from the first “Automotive Software Development Meetup” with Christian Fraas and Michael Fait in Stuttgart last week.

Automotive meetup

On Tuesday, Nov 23rd 2022, we had our very first Automotive Software Development Meetup in the Thoughtworks office in Stuttgart.

For this event, we had 2 speakers, Christian Fraas from Microsoft and Michael Fait from Thoughtworks. Unfortunately I only took notes during the talk from Christian. Michael’s talk was about Rust in automotive.

Christian - Automotive Control Plane

Christian Fraas
1.5 years at MS, before 13 years at Daimler
MS: “why dont we bring cloud ideas and technology in to the car”

What is a control plane

in the cloud
- eg. the thing that orchestrates your network
- does the magic in the cloud
here: Kubernetes Control Plane
- the thing that controls what’s happening in the K8S cluster
- “the magic that happens in the background”
- https://kubernetes.io/docs/concepts/overview/components

Features of K8S

description of your deployments
standardized API
workload orchestration
inventory of deployed workload
extensible (custom resources)
giant community and ecosystem
…

Motivation

it would be great to have all these features above to deploy software to a car
can we bring these concepts into a car?

In-car architecture

there are low level ECUs (micro controller) [majority] in a car, mainly running on Autosar classic
there are high-performance computers (micro processors) [minority] in a car, eg. head unit, telematics unit

Automotive Control Plane

multiple runtime technologies in a car (from low level ECUs to Docker containers)
each of the workloads that go on such a device are deployed in a different way
- it depends on the technology how you can update the software in the car
this variety can lead to a big mess
- eg. resolved by a variety of “How Tos” that explain how to flash a car
it would be nice to have a technology-agnostic way to deploy SW to a car
- extensible for different SW deployment technologies (eg. replace the container runtime at a later time)
- unified configuration of different technologies
expected capabilities
- update
- rollback
- monitoring
- delta update
you need to be able to do that in an environment where you don’t have any connectivity
- eg. during the car being parked in a connection-less parking lot
- crucial requirement
Possible architecture: an update manager on top of an K3S control plane
- reason: K8S has no dependency management for deployment procedures
Use K3S instead of K8S to get a smaller footprint (spoiler: it’s still too big)
Interesting part: how to bring the technology in the systems that are not native K8S (custom resources vs. virtual kubelet)
- possible solution: virtual kubelet makes a facade for the API of your target device (eg. Autosar Classic / Adaptive)
- node (ECU) then acts like a K8S node
- a virtual kubelet allows you to update an Autosar ECU just like a Docker container

Eclipse Leda

all this is “Eclipse Leda”
allows for developers to run their application in a virtualised environment
comes with a vehicle update manager
self-update agent for FOTA
collects metrics and logs and ships them back to the cloud with open telemetry

Demo

Leda is very basic operating system
- Container runtime
- DNS
- CAN drivers
- connectivity
- everything is running in Docker containers
using a virtual machine running in the cloud to emulate an ECU with QEMU
- within this double-virtualised machine, an automotive control plane is running
all tools that work with K8S could now “work in the car”
- example: show all software running in a car with k9s
deployment yaml looks exactly the same for in-vehicle software as for cloud deployments
Eclipse Velocitas as and end2end development toolchain

Challenges for an automotive control plane based on K3S

resource footprint is too big for in-vehicle computers
boot time for K3S is too long for a car
offline update doesn’t work out of the box
self-update doesn’t work out of the box
workload distribution in a car is really static
- K8S features for this is not required

Questions

how does “developer first” fits into the “embedded developer world”
- do embedded developers really want to deal with Kubernetes?
will the concept of embedded developers and cloud developers will merge?
- group of developers will further differentiate
- K8S might not be the tool a embedded developer has in mind right now
- but anyway it will help them develop their software

Further references

Software defined vehicle - Eclipse Foundation

Meetup: Why you should avoid platform monoliths

Wed, 23 Nov 2022 20:00:00 +0000

These are my notes from the Thoughtworks meetup on Nov 23rd 2022 (link contains recording) with Marion Bruns and Rahul Punjabi about platform monoliths and why you should avoid them.

Why you should avoid platform monoliths

Some background

platform as enabler for enterprises to deliver customer value quickly
- in this talk, focus on developer platform
why invest?
- t2market, cost efficiency, reliabilit, security
reality check for dev platforms

“have you faced one or more of these challenges?”
- platform becomes a bottleneck
- low adoption rate amongst developers
- insufficient performance
- new features take long time to be implemented
- costs are rising

online poll reveals: “some of the points above applied to 57% of the audience”

Why do platforms fail?

treating the cloud like on-prem DCs
- delegated to a team that previously maintained the on-prem DC
- ticket-based
- silos
- no interaction with outer world
- no product thinking
- “it takes more to enable autonomous teams and the business value you want to achieve”
not planning for right scale
- scaling up AND scaling down (starting with tons of resources, not really required)
assuming what platform customers want
- “we wil build something and they will come” -> does not really happen
- lack of collaboration with future customers
- lack of product thinking
unbalanced capabilities across teams
- things get developed in isolation
- when times come, devs need to be onboarded without proper capabilities
- platform tends to be too complex

Traditional drivers: monolithic platform formula

seen by a lot of clients, when first approaching TW)

what do we mean by that?
- “platform team box in the cloud”
  - users and access key, IAM, CI/CD, networking, logging, monitoring
  - compute orchestrator (K8S or EC2 instances)
  - “if you want DevOps, Jenkins is probably the first step”
  - underlaying: runtime cluster of nodes
  - service mesh in the middle between cluster and customer workload… service discovery, circuit breaking, service2service comm…
  - client workload running on top, eventually “soft” isolation of workloads
- issue with service mesh - ask: do you really need that? only makes sense with micro services at scale
  - often get implemented without actual need, very complex
  - service meshes can slow down entire architecture
- last but not least: data as the most sensitive asset of your company
  - data is part of the “platform box” - data is sitting there… customer data, product data, potentially highly confidential
  - sits in the same box that is accessible by everybody
there are some use cases where this box-approach can work
- but many times, this approach does not work
centralized platform IS NOT a sensible default
- security
- reliability
- maintainability
- performance

Why is the this monolithic platform formula NOT a sensible default

security incidents
- ask: “when and how they might happen” not “if they may happen”
- think about the attack surface, how might an attacker attack the platform
- what permissions do your roles allow?
- how and where is your data stored - how granular are permissions to data…
share multi-purpose platform
- one container orchestrator vs. workloads of varying characteristics that should run on the platform
  - example: one CI/CD system for all customers, but runners might have completely different characteristics in terms of platform requirements
- capacity management
  - despite automated measures in the container orchestrators, you will be flooded with requests for fixing resource issues
- you lose the capabilities of modern cloud in terms of scalability…
Cluster administration with an enormous failure radius
- how much effort do you have to put into patching and upgrading
- what about the likelihood that an individual workload can block your cluster from udpating appropriately
- what about your confidence in recovering from cluster-wider failures?
Multi-tenancy limitations
- maintaining multi-tenancy on compute, storage and network is complex
- there are a lot of measures that can be taken & provided by cloud provider, but implementing them can become very tedious
- you easily run into an amount of policies that are hard to maintain on a platform level
- limitations in network resources (IPv4 addresses, ports…)

What are the solutions at hand?

What can we learn from Microservices, Data Mesh & Evolutionary Architecture (applied to developer platforms)

Architectural Quantum
- independently deployable component with high functional cohesion, required by system to function properly

Formula for the design of a decentralized platform

Dedicated architectural quantum for each customer
- Azure: one subscription for each customer
what’s inside
- compute
- network integration
- observability tooling
- …
- IMPORTANT: capabilities depend on your use case
how is this benefitial
- security incident: fix it for one affected customer
- setup auto-scaling suitable for the one workload / use case of customer
- you won’t run out of resources for an individual customer (as long as cloud provider can offer more)
- you can apply horizontal scaling (if needed)
shared services in platform
- hybrid networking
- inter-quantum networking
- internet connectivity
- DNS and service discovery
- compliance automation (making sure, users follows policies)
platform services (will vary from context to context - this is just a spectrum)
- IaC and pipelines for infrastructure management
- compliance and government policies
- developer portal
- community forum
we encourage to put as many capabilities as possible in the quantum, since it is still managed by the platform team
Outcomes of running decentralized platforms at scale
- applied to multiple public cloud providers - approach worked for multiple clouds
- how long does it take: productive customers within the year of inception
- allows for self-service for the teams
- allows for security guard rails
- increased cost-transparency

Core capabilities and practices for successful platform strategies

it’s more than just implementation - eg. balance capabilities across teams - how do I approach in a way that everyone can benefit from it
Product Thinking
- do user research, get requirements from your customers, do enough research to get those requirements
- Define measures of success - you have to do this right from the start - are you helping your customers or are you getting in their way?
- product strategy - product owner who can aling with business objectives - have a platform to serve the business objectives
- establish and communicate service level objectives
shared responsibility
- mapping technical capabilities and interfaces (interfaces in terms of “how do you collaborate”)
- what does it mean to define technical capabilities
  - define collboratively the needed tech capabilities
  - define model on how you offer capabilities (just documentation, code snippets, …)
  - define the interaction model with the development teams (self-service, developer portal, …) - but even close collaboration with the dev team
  - how do you want to onboard? you might even need to go into the team(s) and help them with onboarding
self-service
- invest in a developer portal (eg. Backstage)
  - step-by-step guide in how developers can make use of services
  - there should be no need for devs to write tickets to get something (onboarding new user, new VM…)
- build automated, dev friendly interfaces
- provide paved paths for frequent user journeys
- build in quality gates and guardrails
Lightweight governance
- enable autonomous teams!
- treat principles and guardrails as living artifacts
- use compliance as code and automation to enforce them
- continuously collaboration on those principles - allow for users to work and aling on these principles - collaboration is key

Some of the challenges

Isn’t this more expensive?
- return question: how are you looking at costs?
- consider hollistic view of costs over local optimization
- when slowing down customers with monolithic platform eg., experimentation, platform reduces competitive advantages of your dev teams
Orchestration of dependencies across platform infrastructure
Balancing infra capabilities across platform and customer teams (especially when starting with the platform)

Takeaways

monolithic platforms can restrict its own evolution and restric the customers

References & resources

Curated Q&As from 'Trunk-based development Meetup'

Thu, 21 Jan 2021 21:55:29 +0000

Last week, I had the pleasure to speak about Trunk-Based Development together with ThoughtWorks’ Chris Ford and Kief Morris. This blog post contains a curated list of questions and answers from the discussions we had during the meetup.

For those of you who missed the talk, here are the slides from the presentation. The recording of the talk is currently in review, I will link it as soon as it’s available.

But we do trunk-based development with pull requests

There was some irritation around “my definition” of trunk-based development

But we do trunk-based development with pull requests!

The short answer maybe is, that even if there are trunk-based development approaches where you have pull-requests in place, we generally try to avoid this for the sake of faster feedback and fully automated continuous integration workflows.

I disagree with the definition. You can do trunk based with pull requests… my definition:

A source-control branching model, where developers collaborate on code in a single branch called ‘trunk’ *, resist any pressure to create other long-lived development branches by employing documented techniques. They therefore avoid merge hell, do not break the build, and live happily ever after

Does not mean you should not do pull requests

CI/CD with feature branches

On the topic of how feature branches complicate your CI pipelines

Regularly rebasing the feature branch from the main branch would avoid this late feedback problem.

The answer given in the chat / shown on the slides: No, as soon as you have 2 (or more) feature branches, you don’t have continuous integration (because you don’t have the changes of the other feature branches in your branch).

Team size & experience and trunk-based development

How do you integrate less experienced developers in the team who might not yet be familiar with producing good quality code?

Some of the answers:

We use pair programming for knowledge-sharing and supporting our junior team members
We generally provide a safe space to make mistakes and strive for a blame free environment
Don’t misinterpret “pair programming” with “not letting them work on their own” though!

What is a size of the typical team and what is an average experience level (Seniors vs Mids vs Juniors)? Thx.

We have teams of about 8 people with about 4 developers
We always look for a good mix of senior and junior developers - as well as gender and diversity in general

Code Reviews with trunk-based development

You probably will go into how to do effective reviews using trunk based dev, if you don’t have the luxury to pair program all the time, right?

we actually do pair programming all the time, it is an essential part of our development workflow
other than that, we have tech huddles or sessions where we only talk about crucial parts of our code

I believe that the PR with code reviews brought a huge quality improve. It gives the team (not only to your pair) the option get to know (and agree on) a feature implementation? Having a code review at the end would remove an easy option to agree/disagree on how a feature is implemented (as you might have commits from multiple features).

I think the problem lies in the “at the end” here - PRs move the feedback to a very late point in your development workflow - we try to enable this feedback earlier with the tools mentioned in the presentation

Do you have experience with good methods to do reviews more asynchronously together with TBD? I am thinking especially if the team members are remote and potentially in different time zones and pair programming is not as simple from a logistic point of view.

Do you have more examples of code reviews in TBD outside of pair or mob programming?

I think Kief mentioned a few approaches, e.g. make them part of the review or have dedicated sessions for review.
With crucial code, we sometimes make it part of our daily tech huddles
(from a participant) we are using both approaches in our company. The same people commit and push code more often when working with feature branches. Code review is actually a huge problem for us when doing TBD; no context switch for code author, but reviewers have to stop their work
(from a participant) I remember reading (I think on Peter Seibel’s Twitter?) about how in one team they would review code after it got merged to the main branch. Note that this kind of review emphasizes knowledge sharing as opposed to preventing errors.

Feature Toggles

Do you always work with feature toggles when you commit directly into the mainline?

I would say, it depends, e.g. on the maturity of the project. At the beginning of a project - without any production users around - you probably don’t use feature toggles. Later on you might consider them to guarantee a stable prod environment.

Hm, so does it mean that with TBD I have less than a day to implement a feature for the customer? But with, say Git Flow, I can work on the feature for longer period of time?

The short answer is people usually have a “feature toggle” that can turn the feature off until it is all done.
You can still have multiple days/weeks to develop a feature - feature toggles for the win
The value of TBD and toggles for work that takes longer is that it forces you to have discipline to keep the code clean and working. If I take days to work on something without committing, my code tends to get very messy

Feature toggle are good and valid solution. But also, here you can get into a similar hell like with the branch-hell! Not every time the features are completely independent, so you need test different combinations of the feature-toggle-states. Managing these could very complex☹.

As with anything, good design, good discipline is essential
For further details on how to avoid feature toggle hell, refer to this blog post by my dear colleagues Paul and Steve

I think my biggest issue with TBD is that it forces you to use feature-toggles. Using them leads to tech debt.

I wouldn’t say it forces you to do features toggles. It is just that they make a lot of sense with TBD.

If you want to commit often you will come into the situation where you need FTs, so it forces you.

Depends on how you plan your feature road map, imho
Not every commit introduces a new feature, even in the most aggressive greenfield project, I suppose?

Trunk-based development and very large teams

Have you ever tried this with 600 developers pushing to master?

I think if you have 600 developers working on a single codebase you have a problem!
Google does trunk base development with a lot moe programmers ;)
Unless you are google or Facebook, which do this, but they have sophisticated tooling
In most organisations with 600 developers, the software is architected so not all 600 are editing the same parts of the codebase

Mixing trunk-based development with pull request workflow

Do you have experience with a mixed model, where you do mostly trunk-based development, but sporadically accept pull requests from “untrusted developers”?

we do that in our project, which consists of multiple squads. Whenever we make changes in other squad’s repos, we send them a pull request and ask for review, since we are not 100% familiar with their code
still, within a squad, we work trunk-based, since the committers are “trusted” team members

Life span of feature branches

I would say, it all comes down to the question: how long does your branch (change set, PR) live. If it is more than a couple of days, the more you diverge from trunk (and also the change set becomes larger and larger) – therefore you should aim for small and short lived PR

But the time is not the factor. It’s the amount of changed code that matters

I agree to the short-living branches
I partially agree on the dependency on the size of the change, but at the end, the time is more crucial, since the more time passes the less the chance that the developer still knows what she did many days ago and hence there is a bigger effort to make the required changes

Shifting to trunk-based development

Do you have a theory of why most people get so defensive about the feature-branch-based workflow (at least that’s my impression)? It seems to me that the benefits of trunk-based development have been very well-stated for long time, and well-suited for most (at least closed-source) situations already, but it is much less widespread than PRs.

I think people are familiar with how they’ve learned to work. It’s very hard to change, especially when you see, as Michael is saying, there is so much other things you need to change in how you work. So it seems impossible, or silly, or unrealistic to many people. I remember before working in a team that really understood how to do TDD, I found it too hard to do. And I wasn’t sure if it was something people did in real life, or just a nice idea. Then I joined TW, and saw teams doing it for real, and it was mind-blowing

Trunk-based development and QA

Does manual QA have a place in this approach?

Yes, we have the role of a QA that does exploratory testing on the running software

How do you handle legal/industry regulations that require a formal review (more formal than PairProgramming) before comitting to main?

we did not talk about this, but I would say that it often breaks down to traceability, i.e. connect a SW change to a requirement and provide test information and formal

Database migrations and feature toggles

How would you handle database-migrations with feature-toggles? Especially when there are data-migrations wich can’t be reverted.

You need to split the DB-Change into 2 compatible DB-changes. First only add Elements and after feature toggle is changed in prod. 2 DB Change removing the not anymore used old elements. Typ-changes -> introduce new column in step 1 and trigger updating the new column. And in step 2 remove the old column and the trigger
generally you have to plan ahead for non-breaking changes, e.g. provide old features in parallel and phase them out after new features have been successfully introduced
The mind change for DB-developer is huge. As the goal for a for DB-developer was to have only a clean DB structure in Production. And the solution is here to provide this intermediate DB-structure in Production.

Kent Beck’s “test && commit || revert”

Another, maybe somewhat related, more extreme workflow - have you heard about Kent Beck’s “test && commit || revert” If yes, do you have any thoughts about it?

I personally didn’t know it, but thought, I’d link it here.

Team situations around trunk-based development

How would you handle the situation when you have head of dev who draws part of his identity from being a hard and pain in the a** reviewer and therefore hates TBD? Any idea of opening up someone like that to the idea?

if someone draws their identity from being a pain ita, it might be really hard
someone recommended to better run
Trust is important. If your lead doesn’t trust the team that’s an issue :-/

Pair programming and trunk-based development

Pair programming is not an argument for tbd. You can do pair programming also on features. Reality is often more than one dev works on a feature branch

it’s the other way round - I would argue that pair programming is a prerequisite for TBD or at least makes it much easier / facilitates it. You can easily do pair programming with Merge Requests…

Doesn’t convince me. From experience people are different, some like pp some not. With this you’re forcing people into one way of producing good code.

might also be a question of how you approach people in pair programming. I worked with many different “personalities” in a pair programming way and it requires some communication skills and lots of empathy to make everyone comfortable with it…

Trunk-based development and code history

How its source code history affected with TBD in your experience? (rolling-back features, understanding an implementation without documentation, etc)

What would you say about the fact that the source control history gets scattered. Especially if you have serveral people working on different features on the same code base and commiting several times a day. Looking at the history you see a lot of commits. not necessarily connected to each other.

Is this just a question of tooling? Do you see that as an issue that it is harder to grasp the context of changes; does traceability suffer?
During the meetup we talked about how proper tooling makes it easier to group changes in your repository and link it to the underlying story.

References and resources mentioned in the meetup

Curated Q&As from 'Getting Started with Terraform Pipelines'

Thu, 07 May 2020 21:55:29 +0000

Today, I had the pleasure of being the speaker for my very first ThoughtWorks meetup. The topic was “Getting started with Terraform Pipelines” and due to the current (Corona) situation it was an online-only event. You can find the slides here: https://github.com/michaellihs/meetup-2020-05-07-terraform-pipelines/blob/master/presentation.pdf. You can find a recording of the meetup in this link.

This post is a (curated) summary of the questions and answers that were written in the chat, since I think they contain a lot of things to be learned about Terraform and infrastructure as code.

Will you be doomed if state gets somehow lost? Can you recover from a state loss?

you should store your state in a high durability place, such as S3
you are not exactly doomed, but close enough - terraform import can help
- Terraform gives the possibility to import existing ressources into a state
But there is no way to import all resources yet, it is per resource (at least last I checked)
not every resource can be imported
You should take care of import. If the provider is implemented badly it won’t help you very much
you can import existing infrastructure into a state
You should backup your state file aswell :-) maybe work with snapshots before/after apply
the state in the end is a JSON File. If everything else fails, things can fixed by editing the file and reuploading it, although this is a recipe for pain

Does terraform give a graphical representation of the infra getting created like for e.g cloud formation designer provides?

This blog post contains a tool that generates a graph from your Terraform infrastructure

What happens if resorces were changed outside terraform?

Terraform will change them back to what is described in the configuration
- Sometimes this means destroying and re-creating a resource

Followup question: What if your state loss happens because a state apply failed and you have an intermediate state? When a state apply fails because AWS rejects the request?

terraform usually stores backup file while apply

Do you persist the state then?

the state needs to be persisted in any case. Because if you don’t two different people applying terraform code would recreate resources

On which service does terraform run on?

Important to mention it doesn’t run all the time - it’s not a service or something. It is a simple binary that can be run on any command line, e.g. run in a pipeline, makes changes and exits
like this, you can easily run it in any pipeline on any CI/CD server

Is there a possibility to snapshot tf-state in a storage-account?

for Azure: generally yes, see this documentation
- for Azure, there seems to be an open GitHub issue for this
Recommendation for state version is to enable it on the remote storage, e.g. S3 supports versioning of files
terraform cloud is free for state storage as well
usage of Terraform cloud might be dependent on the privacy policies of your company / client though

How do changes to resources, which were triggered manually, affect the terraform state?

if the resource was manually created, it is not represented in the Terraform state at all
if it was created by Terraform and then manually changed, the next Terraform apply will bring it back to the automatically created state

Which systems (providers) are supported by Terraform

These are the supported systems that Terraform can configure: https://www.terraform.io/docs/providers/index.html

tdd is kinda impossible with terraform

terratest is ok to use, but doesn’t help much in tdd
It depends on how you consider TDD, my team have written tests and then written the terraform to deploy working code to make the tests go green - this is out of scope of this talk but we have run talks on infra testing before and will again

Terraform vs. Pulumi - which is gaining more traction these days?

terraform has a bigger offering of providers, pulumi is in very active development as well
pulumi is gaining traction
if you are starting from scratch I would encourage to check both and see which one fits your use case better
My team have focused on Terraform as it’s a common language and tool set to work with Github, K8S and AWS

What does “Go for immutable infra” mean exactly? Other than a DB

a DB is the opposite of an immutable piece of infrastructure, since it mutates over time
immutable infra would be deploying infrastructure that is not changed until the next deployment. For instance, all the networking is based on the TF code reflecting the end state.
another example would be a serverless function or a container. You don’t update or patch it in place, you just replace it every time with a completely new instance when it gets deployed

What does it mean to split state? Is this an outcome of infra modularity?

splitting state means that you have different state files for different parts of your infrastructure
- e.g. this can mean that you have a different state file for each environment (DEV, TEST, PROD) or even for each service’s infrastructure
- messing up state in a DEV environment does not effect PROD

should we split state for each environment?

Where do you run terraform code, in azure devops, azure container or azure shell, or local laptop?

basically you can run it from every place that can run a go binary
For my team we run all terraform code in pipeline only, we do not run terraform locally at all and have not creds to do so. My team make an active choice to do this. Our CI is GoCD running in K8S containers

Is the concept of pipelines and stages applicable with other tools than Terraforn? E.g. with Ansible?

yes, though the declarative nature of terraform tends to make describing the end state a bit easier
Ansible makes much more use of patching resources or configuration in place
This makes staging in general rather hard

It looks like your terraform code for many services is all in a single “infra” repo - was there any intentional decision to do this instead of putting the terraform scripts alongside the service code inside each services repository? What drove that decision?

we had infrastructure in place that we needed to preserve. Starting with what we had and refactoring in small steps seemed like a good approach.
Generally you should split your infrastructure in smaller junks and test and apply them seperately
Still seperation can raise new issues, like dependencies and remote references

do you use atlantis?

We’ve just started using Atlantis - it’s nice :D
Not yet, we are considering it in the future

Do you use HachiCorp Vault for secrets or some parts of CI server for secrets?

Currently we are using K8S “secrets” we will be moving to AWS secrets store “soon” and then in a while we will be standing up vault
We are using the Azure keyvaults, but only use it for deployments, not for runtime, since it seems to be too slow for “real time access”

What’s the best place to version control infrastructure code, in repo with application code or in separate repo?

I think the answer is “it depends” and it also depends how tightly linked your code and infra are

Did you discuss splitting state even more, e.g. one statefile per service? Why did you decide against it?

we considered to do so, but agreed against as a first step, mainly to keep things simple and to have an easy orchestration of our bootstrapping process

monorepo vs micro repos with code?

see above: it depends on the use case
as a rule of thumb: smaller is better

Do you use tools like test terratest, kitchen-terraform, validate? or some part of GoCD can help with testing?

My team currently tests with tflint and terraform validate, we have additional in pipeline tests to ensure resources are available and configured correctly.
We are also looking at awsspec for testing

Is there anyone who experienced with Jenkins for CI pipeline? How good is Jenkins about Terraform CI pipelining?

works good
any CI tool can be used to be honest, though I’ve seen many centralized jenkins instances that are exceedingly hard to use

I see a lot of azure specific configuration? I was expecting something more platform independent so you can describe the state independent of where it is applied (Azure, AWS, etc)?

the examples were taken from Azure, since I am currently working with it
you cannot use Terraform to abstract the cloud provider API and it would make no sense, since the cloud resources strongly vary from provider to provider
Terraform scripts are always cloud specific

Have you looked at terragrunt at all? It can handle splitting and templating of environments. Well worth a look if it fits your needs

+1 to terragrunt, can be used effectively to address some terraform pain points
Terragrunt is something that we are looking at next week to fix some of the pain of copy pasted variables
I think terragrunt will be obsolete in some time as tf adds those teatures

Why not to split states down to each independent servece?

Why not split to each service: There are tons of interdependencies you can create between services and passing variables back and fourth becomes a PITA
My team uses both terraform data calls and terraform remote_state as data sources to pass information between small pieces of terraform code

how “cloud specific” is the resulting terraform code? I’m asking since my team might have to e.g. provision k8s and other services on multiple cloud service providers.

very specific, since resources change by clouds
The k8s terraform will apply to most versions of k8s, but different cloud providers provide different resources as Ashish says
The concepts however are always the same, resources / data objects same templating engine etc
If you want to create a kubernetes cluster, just take a look at the differences between the terraform resource “azurerm_kubenretes_cluster” and “google_container_cluster” … you have to handle ALL those differences yourself

did you setup the pipeline(s) for terraform right away or after your infra reached a certain state?

My team went straight to pipeline never ran it locally. But we where starting with a Greenfield setup and we had the opportunity to make a clear choice of pipelines only
We have sometimes needed to do interactive terraform but only a few times in the last 6 months. We do this from the CI/CDs agents
We actually tag all our infra with the unique pipeline ID as well so we can trace AWS -> pipeline -> code

Where/how do you publish the various terraform packages? Is it just pushing a docker image to a registry every time?

For modules, you can run a terraform registry or just use git but we are not at this stage yet
when it comes to the Azure DevOps pipelines, you can use the pipeline artifacts to store the packages. But those are not (only) modules, but also binaries, scripts and Terraform plans

How do you build the pipeline itself? Also with terraform?

We actually use yaml to define them in git and then have good sutopull them
There is a GoCD terraform provider but it’s a little trickier to automate for us
We’ve actually created AWS CodePipelines with Terraform, which then later runs Terraform.. works
we build pipelines with terraform using aws codepipeline and codebuild
Currently the Azure DevOps Terraform provider is being developed. So for the moment it is not stable and a lot of resources within Azure DevOps unfortunately can’t be terraformed.

Can we elaborate more on why “the download terraform package” is needed in every stage?

In Gitlab CI/CD you can use “artifacts” to store the package…
we do this, since many modern CI servers have stateless stages. This means that if you want to pass anything in between stages, you have to persist it to some external storage on the producing stage and then restore it again on the consuming stage.

What if the “test” stage in prod fails?

That’s an interesting question Steffen, and your team should decide if it’s an alert etc
we don’t have any automated rollback in place and it might be hard to achieve with infrastructure code

When automating terraform plan/apply in a pipeline (no manual approval), do you put any restrictions on? (i.e. don’t automatically apply if there are any destroy’s in the plan)

since we went with a gated approach, we don’t do this
using locks might be a step in this direction

Are you running the pipeline from non-master branches? If, what happens there?

We always run from master
At ThoughtWorks, we have a strong opinion on branches and mostly go for trunk-based development

do you run azure rm templates from teraform/terragrunt?

sorry, I have no ideas what this is
so there is a way to provision resources via powershell modules/integration with UI via Azure resource templates (similar to cloudformation on AWS), when you develope infrastrcuture with teraform you are not always have chnages from Azure in modules

How do you know which version or git commit of the terraform scripts was used to build the current prod?

We tag things with the pipeline ID which is made up of the git short hash + pipeline run count
We also do some AWS tricks to have the pipeline ID embedded in the AWS role assume process that we use
Most CI systems provide the commit hash of the repository for the current build. I.e. you can just tag a revision with the date and environment it’s been deployed to

For “proper” unit testing with all dependencies in memory, it’s looking compelling on prem by combining Terraform with the golang vCenter vcsim tool where you can replicate a vCenter set of resources locally. Is anyone aware of such simulation tools for any of the public clouds?

There are some AWS stub tools, I’ve never looked at them
take a look at localstack

What exactly caused your complete infra outage?

Basically applying a new resource against a new (wrong) terraform state.

How are you guys managing state migrations? For example, I want to rename a module? I want to rename a module module.x.something to module.y.something. We’ve always had to do this “by hand” before our pipeline runs with terraform state mv. Any hints on how to get this in a pipeline in a “migrations”

tf state mv works well
terraform state mv works fine, but its always outside of our pipeline. We are adding it to our PRs right now to say “Hey, before we merge this PR into environment, do these state moves”

But how to “terraform state..” when you don’t have access to the state / infra, because only your CI has

i did it manually, not in ci
that then, of course, breaks the concept of only CI has access
use a temporary one-off pipeline to do such renaming

what about rollbacks with terraform?

no answers provided

Lessons learned from 2 Remote Meetups

Mon, 06 Apr 2020 21:55:29 +0000

During the last 3 weeks, we had 2 remote meetups from our DevOps Stuttgart meetup group. I admit that this doesn’t make us experts for this format (yet), but still there were some learnings, I want to share with you.

Keep it interactive

use the chat (Zoom has one, YouTube has one, Google Meet has one)
ask questions to be answered by the audience via chat
- start with the question where your audience comes from
- prepare some quick polls on the topic of your presentation
have some “virtual social time” after the presentation
- many people might leave the call after the presentation ends, but it’s worth having a discussion / some social time with those who stay
if the presenter answers a question, ask the questioner whether his question has been answered (via chat)
ask for feedback and improvement towards the end of your meetup

Keep it simple

do not use multiple chat channels within the same meetup
- at the beginning, we had YouTube, Meet and Slack - that was to much to keep track of
- use a single chat channel (e.g. Zoom chat or whatever is closest to your conferencing tool)

Dedicated Moderator

help the presenter to focus on the presentation and the talk
collect questions that come up on the chat
- write them down on a document and try to cluster / order them before asking the presenter
- use questions every now and then to make the presentation more interactive
- answer the people asking questions that you got their question and they’ll have a chance to ask it later on
if your meeting is hi-jacked, remove unwanted people from the call

Avoid Zoom Bombing

unless you want unpleasant interruptions during your presenation, don’t publicly share your Zoom links
use passwords for your Zoom sessions and share them via meetup messages sent to your participants
as a host: start sharing your screen the moment you start the zoom call to avoid others share unwanted content
make at least one other participant a (co) host of the meeting, to enable others to handle “unwanted guests” and mute / unmute others

Don’t be afraid of virtual Discussions

at the beginning we were afraid of too many unmuted people joining the discussion
- experience shows: there is no need to be afraid, fewer people will talk / ask questions then during usual meetups
- embrace participation and questions from your audience

Do it - it’s a great Experience!

Even though we ran into some issues, the overall experience of doing remote meetups was great!
Don’t hesitate to bring your meetups online - there are new learning opportunities!
The overall perception of the audience was quite positive and we had a better now-show rate than with our last “real meetup”

Hope this helps to get you started and motivates you to start your own virtual meetups!

Please share feedback and your own lessons learned :)

DevOps Meetup Stuttgart: Remote Working

Thu, 19 Mar 2020 21:55:29 +0000

This is a short summary of our DevOps Stuttgart Meetup from March 19th about remote working. This time we had several speakers presenting in small slots of about 10 - 15 minutes each. For the first time, we streamed the meetup via YouTube while organizing it in Google Meet.

We started with Malte talking about how ZOI became a cloud-first company and how they leverage remote work with using SaaS solutions. Malte also gave a short summary of how ZOI recently helped other companies to move their business operations into the cloud.

Afterwards, I did a short presentation on pair programming and especially the challenges when doing it remotely. Summary:

not only applicable for coding
see it as a chance
be patient
think out loud, express your thoughts clearly
rotate roles & rotate pairs
take breaks
do research individually
don’t continue working on a pair project after working hours
collect feedback and adapt ways of working if necessary

Lena from ThoughtWorks was sharing some remote working practices with us. She explained what basic requirements (e.g. on the hardware side) need to be fulfilled to be successful while working remotely. She also presented some team rituals that her team recently introduced:

Stand-Up to start the day, Stand-Down to seize the day - especially: no work on the project after Stand-Down
Take scheduled breaks, setup an alarm, make sure to drink enough, pomodoro technique
Take care of mental health of your team members
- regular (virtual) coffee / tea time
- lunches together online
- open walk-in room (video conferencing room)
- regular feedback
- (online) game playing together

The last prepared part of the meetup was presented by [Marcus}(https://twitter.com/mcasviper). He talked about how tmux. tmux is a swiss army knife for enhancing your terminal, e.g. by splitting your terminal screen, keeping sessions running while logging off from a server. Or - with connection to our topic - by sharing terminal sessions with other people. Quite a while ago, I created a tmux cheat sheet

Some of the questions being asked during the meetup:

whats your opinion on ms 365/teams as a cloud solution
Can you really compensate human work interaction by (online) gaming?
Have you established a morning / afternoon routine, to connect to each other?
- Morgenkreis, where you meet with your kids in a video call, before you start your (working) day
do you have daily online meeting, like scrum meeting? If yes, how many times do you meeting - once/twice during a day?
- Michael’s team: one standup / daily, 15 min
- Lena’s team: one Stand-Up and one Stand-down. No work to be done after Stand-down
I mean are you only talking about this crisis or about working remote on a regular basis?
Do you use special tools for remote pairing like floobits
can you share some useful tools for working remotely with team?
- see list below
Sounds a lot that the The Five Dysfunctions of a Team has to be solved before pair programming.
- This is best done in person, but do you also do this remote.
Any tips on how to counterbalance uneven power dynamics between pair progr. peers? a shy person might be inclined to give more space to the other person even though he/her has more to contribute.
- I would recommend regular feedback and adapting your ways of working and communication. This might be rather a task for the more extrovert, but make sure to give space for the more silent people to express themselves
Lena, did you have “stand down” before you started working remotely?
- Nope
How do you handle topic like knowledge base? dedicated application? shared drive?
- Confluence or other wikis, READMEs if it belongs to a code base

Some more facts about the meetup:

up to 80 people in the stream
lively discussion on YouTube, Slack and Google Meet
participants from Karlsruhe, Berlin, Reutlingen, Allgaeu, Ludwigsburg, Ehningen, Echterdingen, Oberaichen, Stuttgart, Gerlingen, Landau - Pfalz, Degerloch, Texas, Kiyv, Deizisau, Moscow, Magstadt

You can find the slides to our presentations here.

Suggested Tools:

http://meet.jit.si - video conferencing
https://awwapp.com/ - online whiteboard app
https://www.atlassian.com - Atlassian Suite - Agile planning and Wiki
https://trello.com Agile (team) boards
https://azure.microsoft.com/en-us/services/devops/ Azure DevOps - Full-stack toolsuite for agile software development
http://app.retrium.com - online retrospectives
https://funretro.io/ - online retrospectives
https://scrumlr.io/ - online retrospectives
https://miro.com/ - online whiteboard
https://mural.co/ - online whiteboard

Further References:

DevOps Meetup Stuttgart: Automated Security Testing in Continuous Integration

Thu, 05 Mar 2020 21:55:29 +0000

This is a short summary of our DevOps Stuttgart Meetup from March 5th about automated security testing in Continuous Integration. For the meetup we had Christian Kühn and Arnold Franke from Synyx with us as speakers.

Chris started the presentation with a question who is currently running security tests in their pipelines and I was surprised by the majority of hands being raised. Also it seems like nowadays more then half of the audience is running production workloads in containers.

For motivating the topic of security testing, we’ve been introduced to a recent security incident at Euquifax, where a huge amount of private data (i.e. social security numbers and credit card data) leaked, due to a non-patched Java library. The point is that this incident could have been avoided with proper scanning of dependencies and timely patching.

Afterwards the presentation mainly covered the following topics:

How security is treated in many software projects, a.k.a. “I am a developer, leave me alone with your security stuff, we have a team / specialists for that”
Short introduction to OWASP Top 10 and the CVE database from NIST
- TOP 9 is Using Components with known vulnerabilities which we took a closer look at
Dependency scanning with a “broken” Spring Boot app as an example
- showing that even the simplest Spring Boot application has > 50 transitive dependencies
- so a first important learning of the evening was do not only take care of the code you write, but know your dependencies
Introduce security scans in pipelines - with an example of how this can look like in Jenkins
- distinguish between broken builds and unstable builds. Reason for this is to still be able to ship your artifacts with having vulnerable dependencies in them - as long as you work around the issue within your code.
- Whitelisting depenendencies in dependency security scans allows you to mute findings in your dependencies if they do not impact your artifact (e.g. vulnerabilities in a library that only impact your application when you use a certain feature)
- presented tool for security scans was OWASP Dependency Check
- other tools of interest might be snyk, GitHub offers security scanning for free on open source projects
What are the security-related issues with using Docker images
- Docker images are often based on other images, hard to track versions, if using :latest
- base images are maintained by the community and are usually patched properly
- custom intermediate images are often problematic since people tend to forget about them
- widely used anti-pattern: we have nightly builds of our software and application images but forget to patch intermediate images
Container Scanning - scans Docker images for known vulnerabilities
- presented tool for container / image scanning CoreOS Clair
- there is a Docker image to run clair on demand clair-local-scan
- it is challenging to figure out, which vulnerabilities are critical for you
- there is another tool for scanning images trivy
- hint: run tests in multiple vulnerability testing tools and check which findings are relevant for you
scan your applications endpoints in an API Scanner
- presented tool for automated penetration tests for APIs is OWASP ZAP
- how it works is that a spider first of all generates a list of endpoints for your API that is then passed on to a penetration test
- you can also feed your Open API definition in a Swagger format to make the tool aware of your actual endpoints
- by looking at the tests ZAP is running against your app you can already learn a lot about application security

So in summary, we learned how to add 3 new stages to our Build Pipeline which are

Security checking of dependencies
Vulnerability scanning for Docker images
Penetration testing your (REST) API in the running application

and about the challenges with properly using the three tools and their findings.

After the presentation, someone mentioned OWASP Secure Codebox as another tool you might want to take a look at.

The lively discussion after the presentation showed that it was very well received by the audience. We hope to have further security topics in our DevOps Stuttgart Meetup in the near future. If you want to give a talk, feel free to reach out to me any time.

Here’s the link to the slides.

A big Thank You! goes to the team at ZOI and especially Malte who provided a pleasant location and food and drinks.

Set iTerm badges from command line

Fri, 07 Feb 2020 20:00:00 +0000

When working with multiple terminal windows, it is sometimes useful to have some kind of heading that indicates what each terminal contains. During our last showcase, we used this to let the audience know what each of our three terminal windows is showing.

Here is a short tutorial on how to set this up, so that you can set the badge easily from the command line. We assume that you are using iTerm on a Mac.

Install the iTerm shell integration - if you are using bash, here is all you need to run
```
curl -L https://iterm2.com/shell_integration/install_shell_integration.sh | bash
```
this will add some includes into your ~/.bash_profile. The website contains explanations for other shells as well (e.g. zsh)
Add the following function to your ~/.bash_profile (or ~/.zshrc)
```
function iterm2_print_user_vars() {
    iterm2_set_user_var badge $badge
}
```
thereby you set a iTerm user var badge that reads the content from the $badge shell variable.
reload your shell profile via
```
source ~/.bash_profile
```
In your iTerm profile settings, set \(user.badge)
You can now set your badge in your terminal via
```
export badge='hello badge'
```

Have fun!

Further References

https://technology.customink.com/blog/2016/07/18/how-to-add-badges-to-iterm2/

Socat and Docker

Thu, 02 Jan 2020 20:00:00 +0000

Recently, I learned how to use socat to monitor the network traffic between the Docker CLI and the Docker host. This works by proxying the communication via socat and sending all traffic to stdout. Here is how it works for either Unix sockets or TCP sockets.

Installing `socat` on Mac

If you are running on a Mac, you first need to install socat - easiest done via Homebrew

brew install socat

Monitoring traffic with Unix socket

assuming we have the Docker daemon listening on a Unix socket /var/run/docker.sock

Open a first terminal and run

socat -v UNIX-LISTEN:/tmp/docker.sock,fork UNIX-CONNECT:/var/run/docker.sock

now open a second terminal and run

docker -H unix:///tmp/docker.sock ps -a

You will see all HTTP traffic between the Docker CLI and the daemon in the first terminal.

Monitoring traffic using TCP socket

assuming we have the Docker daemon listening on a TCP socket 127.0.0.1:1234

if you don’t have the Docker daemon listening to a TCP socket, you can first “simulate” this via
docker run -d -v /var/run/docker.sock:/var/run/docker.sock \
    -p 127.0.0.1:1234:1234 \
    bobrik/socat TCP-LISTEN:1234,fork UNIX-CONNECT:/var/run/docker.sock
this will make the Docker daemon listen on 127.0.0.1:1234

Open a first terminal and run

socat -d -d -v TCP-LISTEN:2345 TCP4:127.0.0.1:1234

now open a second terminal and run

docker -H 127.0.0.1:2345 ps -a

Again, you will see all traffic between the Docker CLI and the Docker daemon in the first terminal.

lihsmi.ch

Meetup: CI/CD for relational databases

A short story about a DB Change

What happens if DB changes are avoided

What we know about releasing SW

4 (5) key metrics

High performing teams and delivery capabilities

Goal of SW release

DB development in numbers

CI/CD for apps

CI/CD for DB

5 pre conditions for DB CI/CD

1. Version control

BUT: pure file-based dev is uncommon in DB development

Repeatability of migration scripts

Branching strategy

2. Static code analysis

3. Database migration tools

4. Automated DB tests

5. Decoupled codebase

CI/CD pipelines for databases

Effects of CI/CD in database development

Questions

Meetup: Automotive Software Development

Automotive meetup

Christian - Automotive Control Plane

What is a control plane

Features of K8S

Motivation

In-car architecture

Automotive Control Plane

Eclipse Leda

Demo

Challenges for an automotive control plane based on K3S

Questions

Further references

Meetup: Why you should avoid platform monoliths

Why you should avoid platform monoliths

Some background

Why do platforms fail?

Traditional drivers: monolithic platform formula

Why is the this monolithic platform formula NOT a sensible default

What are the solutions at hand?

Formula for the design of a decentralized platform

Core capabilities and practices for successful platform strategies

Some of the challenges

Takeaways

References & resources

Curated Q&As from 'Trunk-based development Meetup'

But we do trunk-based development with pull requests

CI/CD with feature branches

Team size & experience and trunk-based development

Code Reviews with trunk-based development

Feature Toggles

Trunk-based development and very large teams

Mixing trunk-based development with pull request workflow

Life span of feature branches

Shifting to trunk-based development

Trunk-based development and QA

Database migrations and feature toggles

Kent Beck’s “test && commit || revert”

Team situations around trunk-based development

Pair programming and trunk-based development

Trunk-based development and code history

References and resources mentioned in the meetup

Curated Q&As from 'Getting Started with Terraform Pipelines'

Lessons learned from 2 Remote Meetups

Keep it interactive

Keep it simple

Dedicated Moderator

Avoid Zoom Bombing

Don’t be afraid of virtual Discussions

Do it - it’s a great Experience!

DevOps Meetup Stuttgart: Remote Working

DevOps Meetup Stuttgart: Automated Security Testing in Continuous Integration

Set iTerm badges from command line

Further References

Socat and Docker

Installing socat on Mac

Monitoring traffic with Unix socket

Installing `socat` on Mac