The following example with be alternative to curl http://localhost:8080/health/ready or wget -qO- http://localhost:8080/health/ready using just bash built-in capabilities:

1
2
3
4

exec 3<>/dev/tcp/localhost/8080
echo -e "GET /health/ready HTTP/1.1\r\nHost: localhost\r\nConnection: close\r\n\r\n" >&3
cat <&3
exec 3>&-

Why do we need terminal integration?

Most AI-enabled IDEs use terminal integration to run commands and get the output back in the IDE. This is useful for fully automated tasks, such as running tests, building projects, and more. In some cases, you can do this manually in an external terminal, but you will lose some of the IDE’s automation capabilities.

Be aware!

Blindly trusting AI is dangerous! There have already been cases where AI followed malicious instructions and caused harm. Always check the list of allowed commands for automatic execution, and try to run your IDE in a sandbox to prevent access to unrelated data.

Why can terminal integration freeze?

In most cases, the problem is that using complicated and slow zsh themes (like oh-my-zsh with powerlevel10k) makes the terminal startup too slow, causing the IDE’s terminal integration to time out while waiting for the shell to be ready. Additionally, some themes use special characters that are not properly handled by the IDE terminal.

How to fix it?

Disable the theme. You can do this by editing your ~/.zshrc file and setting the theme to empty (“”), which will disable it and improve terminal startup time.

1
2
3
4
5

if [[ "$TERM_PROGRAM" == "kiro" || "$TERM_PROGRAM" == "vscode" ]]; then
  ZSH_THEME="" # disable theme for Kiro and VSCode terminals
else
  ZSH_THEME="powerlevel10k/powerlevel10k"
fi

This code checks if the terminal is running inside Kiro or VSCode, and if so, it disables the theme. Otherwise, it uses the powerlevel10k theme.

TL;DR

• You need DMARC enforcement to use BIMI
• BIMI requires a specific SVG logo format
• BIMI requires a Verified Mark Certificate (VMC) or Common Mark Certificate (CMC) to display logos
• You need to publish a DNS record with your logo URL and optionally (but in practice, required for most providers) a VMC URL

You can check your BIMI implementation using tools like the BIMI Validator.

Certificates are required for displaying logos, which can be expensive (around $1,000-$1,500 per year). Let’s Encrypt does not provide VMCs due to their manual verification requirements.

No fun here.

How BIMI Works

The magic happens through a combination of DNS records and email authentication protocols:

1. Email Authentication Foundation: BIMI builds upon SPF (Sender Policy Framework), DKIM (DomainKeys Identified Mail), and DMARC protocols
2. DNS Record Publishing: Organizations publish a new standardized DNS record containing a URL to their logo
3. DMARC Enforcement: The mailbox provider checks that the sending domain has a DMARC policy configured with enforcement (p=quarantine or p=reject)
4. Logo Display: If both checks are successful, supporting mailbox providers may display the logo from the BIMI record

The process ensures that only legitimate, authenticated emails can display brand logos, making it harder for attackers to impersonate trusted brands.

How to Make Your Own BIMI

Implementing BIMI for your organization involves several steps:

Prerequisites

1. DMARC Enforcement: Your domain must have a DMARC policy of at least p=quarantine or p=reject
2. Logo Ownership: You must own the trademark/rights to the logo you want to display (for VMC), otherwise you can use a Common Mark Certificate (CMC), which is somewhat easier to obtain but still costly
3. SVG Logo: Your logo must be in SVG with specific requirements

Step-by-Step Implementation

1. Prepare Your Logo:

   • Create an SVG file using the SVG Tiny PS profile
   • Make it square with a solid background color
   • Ensure it displays well in a circle (some email clients crop it)
   • Host it on HTTPS with proper CORS headers

2. Create the BIMI DNS Record:

   1	default._bimi.technicaldomain.xyz IN TXT "v=BIMI1; l=https://technicaldomain.xyz/logo.svg;"

3. Test Your Implementation:
   Use tools like the BIMI Validator to verify your setup

BIMI Record Attributes

• v=BIMI1: Indicates this is a BIMI record
• l=URL: The hosting location of the SVG image
• a=URL: The hosting location of the VMC (Verified Mark Certificate) - optional but recommended, without it the logo will not be displayed in some email clients (for example Apple Mail will not display the logo without a VMC)

Certificates and BIMI

This is where things get expensive. While you can implement basic BIMI without certificates, many major email providers require Verified Mark Certificates (VMCs) to actually display logos.

Paid VMC Options

Currently, only several Certificate Authorities offer VMCs: DigiCert, GlobalSign and SSL.com

Cost: VMCs typically cost $1,000-$1,500 per year - a significant investment just to display a logo in emails.

The Let’s Encrypt Discussion

Unfortunately, Let’s Encrypt cannot and will not offer VMCs. As discussed in their community forums, VMCs require manual verification of trademark ownership, which goes against Let’s Encrypt’s automated certificate issuance model. A Let’s Encrypt staff member confirmed: “Because of the manual verification required for VMC, Let’s Encrypt cannot implement it. I suspect there will never be a free or low cost option.”

This creates a significant barrier to BIMI adoption, especially for smaller organizations that can’t justify the annual expense.

Benefits of Using BIMI

For Organizations

• Brand Recognition: Your logo appears next to authenticated emails
• Enhanced Trust: Recipients can visually identify legitimate emails
• Professional Appearance: Emails look more polished and trustworthy

For Email Recipients

• Visual Authentication: Quick visual confirmation of sender legitimacy
• Reduced Phishing Risk: Harder for scammers to fake branded emails
• Better User Experience: Easier to identify important emails

Real-World Examples

Several major organizations use BIMI successfully:

• amazon.ca: Amazon Canada displays their logo in supported email clients
• m.wealthsimple.com: The financial services company uses BIMI for brand recognition
• skipthedishes.com: The food delivery service leverages BIMI for order confirmations

How to Check BIMI Implementation

You can verify BIMI records using command-line tools:

1
2

# Check for BIMI DNS record
dig TXT default._bimi.technicaldomain.xyz

You will see a response like this if BIMI is implemented:

1
2
3
4
5
6
7
8
9
10
11
12
13

; <<>> DiG 9.10.6 <<>> TXT default._bimi.technicaldomain.xyz
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 42410
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1220
;; QUESTION SECTION:
;default._bimi.technicaldomain.xyz. INTXT

;; ANSWER SECTION:
default._bimi.technicaldomain.xyz. 300 INTXT"v=BIMI1;l=https://technicaldomain.xyz/logo.svg;a=https://technicaldomain.xyz/certchain.pem"

1
2
3
4
5

# Download and inspect the logo
curl -s https://technicaldomain.xyz/logo.svg

# If there's a VMC, inspect the certificate
curl -s https://technicaldomain.xyz/certchain.pem | openssl x509 -text -noout

For a complete verification, you can use the BIMI validator at bimivalidator.authmilter.org.

Mailbox Provider Support (at the time of writing)

BIMI support varies significantly across email providers:

Full Support with VMC requirement:

• Gmail (Google Workspace)
• Apple Mail

Self-asserted BIMI Support (no VMC required):

• Yahoo
• Some other providers

Growing Support:

• Fastmail
• La Poste
• Onet Poczta
• Zone

No Support:

• Microsoft Outlook

The inconsistent support means your investment might not pay off across all email clients. Especially if your organization relies heavily on Microsoft Outlook, which currently does not support BIMI at all.

Conclusion

BIMI is an interesting technology that bridges email authentication and brand recognition. It’s somewhat amusing that we’ve reached a point where displaying a simple logo in email requires complex certificate infrastructure and significant annual costs.

The Reality Check:

• For Large Brands/Banks: BIMI makes sense if budget allows and brand protection is critical
• For SMBs: The $1,000+ annual cost is hard to justify for logo display
• For Everyone: Limited mailbox provider support reduces the return on investment

If you’re considering BIMI, focus first on solid email authentication (SPF, DKIM, DMARC) - these provide real security benefits. BIMI is the cherry on top, not the foundation of email security.

Bottom Line: Funny concept, makes some sense for brands with deep pockets, but costly implementation and patchy mailbox provider support limit its practical appeal.

But if you’re banking on email as a key marketing channel, the investment in BIMI could pay off in increased brand trust and recognition.

Evaluate your needs and budget before investing in BIMI.

macOS

Run the following command in the terminal:

1

sudo dscacheutil -flushcache; sudo killall -HUP mDNSResponder

Windows

Run the following command in the Command Prompt with administrator privileges:

1

ipconfig /flushdns

Linux

For systems using systemd, run the following command in the terminal:

1

sudo resolvectl flush-caches

Alternatively, you can restart the service:

1

sudo systemctl restart systemd-resolved

The end.

TL;DR

You need to set up kafka-proxy with custom configuration and a custom TLS certificate, then expose it via a Network Load Balancer.

AWS Blog references:

• Secure connectivity patterns to access Amazon MSK across AWS Regions
• Connect Kafka client applications securely to your Amazon MSK cluster from different VPCs and AWS accounts

Use Cases

Exposing Amazon MSK from a private subnet with a custom FQDN and TLS certificate is useful in the following scenarios:

1. Multi-Region Deployments: When you need to connect clients from different AWS regions to your MSK cluster securely.
2. Hybrid Cloud Environments: When external clients or on-premises systems need access to your MSK cluster without exposing it directly to the public internet.
3. Compliance Requirements: When your organization mandates the use of private subnets and custom domain names for secure communication.
4. Custom Domain Branding: When you want to use a custom FQDN for branding or easier client configuration.

High-Level Design

The following assumptions are presented here:

• Amazon MSK is deployed in a private subnet. No extra steps in configuration are made.
• TLS certificate and private key are already stored in AWS Secrets Manager.
• Certificate Manager is not used in this case because we need to retrieve the private key.
• NLB TLS listener is not used in this case because we need to replace not only the certificate but also update Kafka listeners’ FQDN, and kafka-proxy does all this job for us.
• Appropriate IAM roles are already configured to provide access to necessary resources.
• Security groups configuration is out of scope.

Step by Step

Precondition

VPC is configured, and you have enough unused IP addresses in your private VPC to run MSK, ECS, and NLB.

Certificates

Put your certificate and private key into a secret in AWS Secrets Manager.
You can automatically get a certificate for free from various CAs with ACME support, for example:

• Let’s Encrypt
• ZeroSSL

Amazon MSK Deployment

Use your favorite automation tool (Terraform, Pulumi, or CDK) or create it manually to get Amazon MSK up and running.

Elastic Container Services

ECS setup and configuration is pretty standard as well.
Here are a few hints:

• Do not forget to mirror all used Docker images to ECR to avoid cases where you are unable to spin up the service due to image unavailability or Docker Hub rate limits.
• To deliver the certificate, we will use a special init container that retrieves the certificate and private key from AWS Secrets Manager and places them in a shared volume.

Below you can find ECS task definitions to bring everything together:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138

[
  {
    "name": "aws-secret-to-file",
    "image": "ghcr.io/technicaldomain/aws-secret-to-file:latest",
    "cpu": 100,
    "enableExecuteCommand": true,
    "memory": 32,
    "memoryReservation": 32,
    "essential": false,
    "command": [
      "/bin/aws-secret-to-file",
      "--secret=/demo/kafka/tls-certificate",
      "--output=/app/certificate.crt",
      "--secret=/demo/kafka/tls-certificate-key",
      "--output=/app/certificate.key"
    ],
    "environment": [
      {
        "name": "AWS_DEFAULT_REGION",
        "value": "us-east-1"
      }
    ],
    "mountPoints": [
      {
        "readOnly": false,
        "containerPath": "/app",
        "sourceVolume": "certificates"
      }
    ],
    "portMappings": [
      {
        "containerPort": 1196,
        "hostPort": 0,
        "protocol": "tcp"
      },
      {
        "containerPort": 1197,
        "hostPort": 0,
        "protocol": "tcp"
      },
      {
        "containerPort": 1198,
        "hostPort": 0,
        "protocol": "tcp"
      }
    ],
    "volumesFrom": [],
    "linuxParameters": {
      "initProcessEnabled": true,
      "maxSwap": 1024,
      "swappiness": 60
    },
    "secrets": [],
    "startTimeout": 120,
    "stopTimeout": 120,
    "disableNetworking": false,
    "privileged": false,
    "readonlyRootFilesystem": false,
    "interactive": true,
    "pseudoTerminal": true,
    "logConfiguration": {
      "logDriver": "awslogs",
      "options": {
        "awslogs-group": "demo-log-group",
        "awslogs-region": "us-east-1",
        "awslogs-stream-prefix": "ecs"
      }
    }
  },
  {
    "name": "kafka-proxy",
    "image": "account_id.dkr.ecr.us-east-1.amazonaws.com/kafka-proxy:0.4.2",
    "cpu": 1000,
    "enableExecuteCommand": true,
    "memory": 512,
    "memoryReservation": 2048,
    "essential": true,
    "command": [
      "server",
      "--http-metrics-path=/metrics",
      "--tls-enable",
      "--proxy-listener-cert-file=/app/certificate.crt",
      "--proxy-listener-key-file=/app/certificate.key",
      "--proxy-listener-tls-enable=true",
      "--bootstrap-server-mapping=b-1.mskname.random.a123.kafka.us-east-1.amazonaws.com:9096,0.0.0.0:1196,kafka.demo.technicaldomain.xyz:1196",
      "--bootstrap-server-mapping=b-2.mskname.random.a123.kafka.us-east-1.amazonaws.com:9096,0.0.0.0:1197,kafka.demo.technicaldomain.xyz:1197",
      "--bootstrap-server-mapping=b-3.mskname.random.a123.kafka.us-east-1.amazonaws.com:9096,0.0.0.0:1198,kafka.demo.technicaldomain.xyz:1198",
      "--log-format=json"
    ],
    "environment": [],
    "mountPoints": [
      {
        "readOnly": null,
        "containerPath": "/app",
        "sourceVolume": "certificates"
      }
    ],
    "portMappings": [],
    "volumesFrom": [],
    "linuxParameters": {
      "initProcessEnabled": true,
      "maxSwap": 1024,
      "swappiness": 60
    },
    "dependsOn": [
      {
        "containerName": "aws-secret-to-file",
        "condition": "COMPLETE"
      }
    ],
    "secrets": [],
    "startTimeout": 120,
    "stopTimeout": 120,
    "disableNetworking": false,
    "privileged": false,
    "readonlyRootFilesystem": false,
    "interactive": true,
    "healthCheck": {
      "retries": 3,
      "command": [
        "CMD-SHELL",
        "wget -q -O - http://localhost:9080/health | grep OK || exit 1"
      ],
      "timeout": 5,
      "interval": 30,
      "startPeriod": null
    },
    "pseudoTerminal": true,
    "logConfiguration": {
      "logDriver": "awslogs",
      "options": {
        "awslogs-group": "demo-log-group",
        "awslogs-region": "us-east-1",
        "awslogs-stream-prefix": "ecs"
      }
    }
  }
]

All the magic happens when we provide the bootstrap-server-mapping parameter. In this configuration, we:

• Connect to b-1.mskname.random.a123.kafka.us-east-1.amazonaws.com:9096 Kafka bootstrap server.
• Map it to the local 0.0.0.0:1196 host and port.
• Update the advertised host and port to the value we provide to the outside (kafka.demo.technicaldomain.xyz:1196).

Then you can expose this ECS via NLB and have the correct custom FQDN and valid TLS certificate.

Job done.

Background

AWS Secrets Manager is a powerful service for handling sensitive objects. In this service, you can store not only database credentials but also key-value pairs, plaintext, and binary secrets. It is often very useful to manage sensitive data (such as JKS, TLS certificates and keys, or SSH keys) in Secrets Manager and deliver it to your workload on demand.

I love this use case, and to simplify and speed up the process, I created a very simple tool to retrieve plaintext or binary secrets and save them to a local file. I mostly use it in sidecars or init containers to bring required data to my workload.

How to Use It

To download a binary secret, simply run:

1

/bin/aws-secret-to-file --secret=/secret/name/here --output=./location/for/the/file --binary

The same approach works for plaintext secrets. You can retrieve a single secret:

1

/bin/aws-secret-to-file --secret=/secret/name/here --output=./location/for/the/file

Or retrieve multiple secrets:

1
2
3

/bin/aws-secret-to-file \
    --secret=/secret/name/here1 --output=./location/for/the/file1 \
    --secret=/secret/name/here2 --output=./location/for/the/file2

How to Get It

Pull the Docker image from GitHub.

In my case, I’ve created a special Kubernetes controller to work with a GitHub App to exchange the App JWT for an installation-specific and short-lived GHS token. This controller updates the token before it expires and updates some third-party integrations such as ArgoCD OCI repository credentials and Dockerconfig JSON secrets. Unfortunately, this controller is currently useless due to GitHub limitations: only personal access tokens (classic) can access private registries (see GitHub discussion for more details).

TL;DR

References to create your own Kubernetes controller:

• Sample controller GitHub repository: sample-controller
• Generators for kube-like API types GitHub repository: code-generator

My implementation:

• GitHub App jwt2token Kubernetes controller: github-app-jwt2token-controller
• Helm chart source code: github-app-jwt2token-controller
• Helm chart public repository: helm

GitHub discussion:

• Using GitHub Apps to access private registry is here

Basic Understanding of Kubernetes Controllers

Kubernetes controllers are control loops that monitor the state of your cluster resources and make necessary adjustments. Kubernetes provides several built-in controllers to manage various resources such as pods, deployments, services, and replica sets. By design, controllers continuously reconcile the current state of the cluster resources with the desired state defined in the Kubernetes resource manifests.

To understand what is under the hood of Kubernetes Controllers, we can refer to an official controller example, sample-controller. This repository provides a comprehensive diagram that helps to understand the underlying components of a typical Kubernetes controller.

You can see that the client-go library covers a lot of interaction and components by itself.

From the custom Kubernetes controller implementation perspective, we can identify two main tasks in the controller workflow:

• Use informers to watch add/update/delete events for the Kubernetes resources we want to know about.
• Consume items from the workqueue and process them.

client-go provides informers for standard resources, for example, deployments: k8s.io/client-go/informers/apps/v1/DeploymentInformer. These informers contain everything needed at the top of the picture, they provide the reflector, indexer, and local storage.

But we are here not for watching some resources, we need to define and proceed with our own.

How I Created a Kubernetes Controller

Here is a high-level plan to achieve my goals:

1. Create Custom Resource Definition (CRD) for ArgoCD repositories.
2. Create CRD for Dockerconfig JSON.
3. Create CRD for GHS tokens.
4. Implement logic to manage ArgoCD repositories.
5. Implement logic to manage Docker configurations.
6. Implement logic to remove expired GHS tokens.

Preparation

Before we begin, we need to get code-generator. It helps us to automatically generate tons of code instead of manually writing it.

Then we need to update the hack/update-codegen.sh script to have something like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

#!/usr/bin/env bash
set -o errexit
set -o nounset
set -o pipefail

SCRIPT_ROOT=$(dirname "${BASH_SOURCE[0]}")/..
CODEGEN_PKG=${CODEGEN_PKG:-$(cd "${SCRIPT_ROOT}"; ls -d -1 ./vendor/k8s.io/code-generator 2>/dev/null || echo ../code-generator)}

source "${CODEGEN_PKG}/kube_codegen.sh"

THIS_PKG="github.com/technicaldomain/github-app-jwt2token-controller"

kube::codegen::gen_helpers \
    --boilerplate "${SCRIPT_ROOT}/hack/boilerplate.go.txt" \
    "${SCRIPT_ROOT}/pkg/apis"

kube::codegen::gen_client \
    --with-watch \
    --output-dir "${SCRIPT_ROOT}/pkg/generated" \
    --output-pkg "${THIS_PKG}/pkg/generated" \
    --boilerplate "${SCRIPT_ROOT}/hack/boilerplate.go.txt" \
    "${SCRIPT_ROOT}/pkg/apis"

Custom Resource Definition

Here, we need to create YAML definitions for all our custom resources:

CRD definition for the resource that be responsible for updating the password field for OCI repository by GHS token generated by the controller:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: argocdrepos.githubapp.technicaldomain.xyz
spec:
  group: githubapp.technicaldomain.xyz
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                privateKeySecret:
                  type: string
                argoCDRepositories:
                  type: array
                  items:
                    type: object
                    properties:
                      repository:
                        type: string
                      namespace:
                        type: string
            status:
              type: object
              properties:
                token:
                  type: string
      subresources:
        status: {}
  scope: Namespaced
  names:
    plural: argocdrepos
    singular: argocdrepo
    kind: ArgoCDRepo
    shortNames:
      - gha

CRD definition for the resource that be responsible for generating/updating docker config JSON with the actual GHS token:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49

---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: dockerconfigjsons.githubapp.technicaldomain.xyz
spec:
  group: githubapp.technicaldomain.xyz
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                privateKeySecret:
                  type: string
                dockerConfigSecrets:
                  type: array
                  items:
                    type: object
                    properties:
                      secret:
                        type: string
                      namespace:
                        type: string
                      registry:
                        type: string
                        default: "ghcr.io"
                      username:
                        type: string
                        default: "x-access-token"
            status:
              type: object
              properties:
                token:
                  type: string
      subresources:
        status: {}
  scope: Namespaced
  names:
    plural: dockerconfigjsons
    singular: dockerconfigjson
    kind: DockerConfigJson
    shortNames:
      - dcj

CRD definition to store all generated GHS tokens:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: ghss.githubapp.technicaldomain.xyz
spec:
  group: githubapp.technicaldomain.xyz
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                token:
                  type: string
            status:
              type: object
              properties:
                expiresAt:
                  type: string
                  format: date-time
      subresources:
        status: {}
  scope: Namespaced
  names:
    plural: ghss
    singular: ghs
    kind: GHS
    shortNames:
      - ghs

Please pay attention, we are trying to create namespaced CRDs to limit access to them in the future if needed.

Then, we need to define types in pkg/apis/githubappjwt2token/v1/types.go to be able to work with defined CRDs. Also, we need to add special annotations in the code to tell code-generator what should be generated for this type. For example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

// +genclient
// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
type ArgoCDRepo struct {
metav1.TypeMeta   `json:",inline"`
metav1.ObjectMeta `json:"metadata,omitempty"`

Spec   ArgoCDRepoSpec   `json:"spec,omitempty"`
Status ArgoCDRepoStatus `json:"status,omitempty"`
}

type ArgoCDRepoSpec struct {
PrivateKeySecret   string               `json:"privateKeySecret"`
ArgoCDRepositories []ArgoCDRepositories `json:"argoCDRepositories"`
}

type ArgoCDRepositories struct {
Repository string `json:"repository"`
Namespace  string `json:"namespace"`
}

type ArgoCDRepoStatus struct {
Token     string      `json:"token,omitempty"`
}

// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
type ArgoCDRepoList struct {
metav1.TypeMeta `json:",inline"`
metav1.ListMeta `json:"metadata,omitempty"`
Items           []ArgoCDRepo `json:"items"`
}

The full implementation you can find here - types.go

Right now we can run ./hack/update-codegen.sh to generate clientset, informers, and listers.
All generated codebases will be located under the pkg/generated path of the repo. And we can go to the implementation logic of our controllers.

Controller Logic

The logic for controller_argocdrepo.go and controller_dockerconfigjson.go is pretty similar and described in the diagram below:

Every resource with kinds ArgoCDRepo and DockerConfigJson in the scope of githubapp.technicaldomain.xyz/v1 is processed by the appropriate controller.
The controller checks the resource status (subresource), in the status recorded md5 of the appropriate GHS custom resource. The controller checks for the existence of this GHS resource, if it exists do nothing, otherwise, the controller calls a special function to retrieve the GitHub App private key, GitHub App ID, and GitHub App Installation ID from the secret (name of the secret defined in the custom resource field).

Then the controller creates and signs JWT based on the GitHub App private key, GitHub App ID, and GitHub App Installation ID, after that the controller calls a special GitHub API endpoint to exchange this JWT for a GHS token. Then calculate md5 for this token, create a new GHS resource, and store inside this resource token and information about token expiration. md5 is used as the name of the GHS resource, also this name is recorded in ArgoCDRepo or DockerConfigJson status field.

After that, based on the type of the resource, the controller looks for all ArgoCD repositories defined in the ArgoCDRepo custom resource and updates the password field to the new token.
For DockerConfigJson the story is about the same, but in this case, the controller generates a docker config.

The logic implemented in controller_ghs.go is much more straightforward.

Here the Kubernetes controller just checks for the resource expiration time, and if it is less than 15 minutes - simply delete the resource. That’s it.

Local Run and Debug

To debug this controller, you should have a deployed and accessible Kubernetes cluster (k3s works, I’ve checked).

To run the controller locally, please ensure that you have updated your kubeconfig and that your cluster is set for debugging in the selected context.

Then simply run go run . -kubeconfig=$HOME/.kube/config. The controller will be compiled and will try to connect to your Kubernetes cluster.

To work with your favorite debugger and put some point break, please refer to your IDE guidelines and best practices.

Installation

This controller can be easily installed using the github-app-jwt2token-controller Helm chart.

Here is an example of an ArgoCD application:

Folder structure:

1
2
3
4

.
├── Chart.yaml
└── templates
    └── resources.yaml

And content

1
2
3
4
5
6
7
8
9

apiVersion: v2
name: github-app-jwt2token
version: 1.0.0

dependencies:
  - name: github-app-jwt2token-controller
    version: 1.1.2
    repository: https://technicaldomain.github.io/helm

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

apiVersion: githubapp.technicaldomain.xyz/v1
kind: ArgoCDRepo
metadata:
  name: example-argo-github-app
spec:
  privateKeySecret: technicaldomain-gha-argo-app
  argoCDRepositories:
    - repository: repo-1114444111
      namespace: argocd
---
apiVersion: githubapp.technicaldomain.xyz/v1
kind: DockerConfigJson
metadata:
  name: example-docker-config
spec:
  privateKeySecret: technicaldomain-gha-argo-app
  dockerConfigSecrets:
    - secret: ghcr
      namespace: gha-token-controller
      registry: ghcr.io
      username: x-access-token

A secret with the name technicaldomain-gha-argo-app should be created in advance using the same namespace where the controller is deployed.
Here is an example of the secret:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

apiVersion: v1
kind: Secret
metadata:
  name: github-app-private-key
  namespace: default
stringData:
  appId: "1111111"
  installationId: "2222222"
  privateKey: |
    -----BEGIN RSA PRIVATE KEY-----
    MIICWgIBAAKBgExrZNCJPDFzjkAzOwlJASNurUkmgzy66yx2tXf1xVZanMN8w5zq
    jQGct+vv3TAXEix5qHG+yMfY0tz2NVEYKcNKjcQnvbDpLtm9qjZgZ2UiDI92civR
    lIGt/0DzXp3Ooq2wxW3rUoIBQE5kBMFdvxJ/QC0zGrKjbQciABTHgj49AgMBAAEC
    gYBJtyauilMIGMHVaBXApS118mMxtvbNdDk60N/H8coDvLCPWiCPkymlrnk0HFMu
    +nJLeKdl4XVoYd01zEIuEbLmYpdafLW5GZ9QtUBno7wVh1js3ahbbwy1+Isj1Xk1
    Wu7nKDbkgYirwb0Ly+CmrieQ+X/Pnm8Z7eyfpPYSDhMQuQJBAJfjfbtAdAuSrS+A
    v9Ti0uqHVzcUeBTF1JWFYnvIRdnx/S4geLdVxmI6Lque5jJdkB2Vmb38fVZzi7Sh
    DKBqT08CQQCAzQgNrpPPhjdn115P9ZAOh5uo7lZFFG+7mc065cZaJyT+PgyuI5pI
    wjdM/Ne/mVSy9eI6GydG+JGWcTOo5BazAkAKBqQ4Bgsi8G2qIw+Gl+pgPMrPAfTj
    OiPMMt/LV+70cfrKXq5ZO7o6paiK/5QmYvKuYT+iwNXtLPdd1vukYyAVAkAaLA93
    4EKOx8IYaq3yZ36nRSz/LbcAAIAXyc/nKOueRBgDRY6EEB34rOZZ0YLxnvGUD9yx
    W/UmObozrLsHlZl7AkBxoq8qqsNkcQtz970QPoh9WSAIk0afOgcWKyKIyeN8Uu2i
    qraqR6mYwgJSPcc1RbOD7ykdMMq03g6upsk4Ws0P
    -----END RSA PRIVATE KEY-----
type: Opaque

Conclusion

Creating a Kubernetes controller is not a complicated process. The official documentation and examples provide a quick understanding of what you need to do and how to do it. Creating your controller and custom resources unlocks the unlimited power of Kubernetes and brings real flexibility in extending functionality.

Unfortunately, my controller is currently limited due to the absence of necessary functionality from GitHub’s side. However, I believe that it will be implemented someday, allowing me to enjoy all the features provided by my controller.

This is a quite common problem with quite common solutions.

Build-time configuration: Create different app configurations and build different artifacts for each of the environments. But in that case, you will get a separate set of artifacts and can’t guarantee that your production application is exactly the same as what you use for testing.

Runtime configuration: Instead of building separate artifacts, you can include some placeholders for all configurations and replace them with actual values during application deployment, like here. This approach is much better from my point of view, but also has some cons. For example: increased complexity in deployment scripts, potential security risks if placeholders are not managed properly, and harder to debug issues related to configuration values.

Runtime configuration with external configuration: In this approach, we can move our application configuration to a separate external file (let’s call it config.json) and fetch this file during application startup. This option is preferable for many reasons: it is easier to manage and update configuration without rebuilding the application, and it simplifies the deployment process by separating configuration from the application code.

Here is a simple example implementing this approach for a React application.

First of all, you need to create the config and put it into the public folder:

1
2
3
4
5

{
    "settingKey1": "value1",
    "settingKey2": "value2",
    "settingKey3": "value3"
}

Then, you need to create a few utils to help you work with the configuration:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

let config = null;

export const loadConfig = async () => {
  if (config) return config; // Return cached config if already loaded.

  const response = await fetch('/config.json');
  if (!response.ok) {
    throw new Error(`Failed to load configuration: ${response.statusText}`);
  }

  config = await response.json();
  return config;
};

export const getConfig = () => {
  if (!config) {
    throw new Error('Configuration not loaded yet. Call loadConfig() first.');
  }
  return config;
};

And, finally, attach the config to your application:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

import React, { useEffect, useState } from 'react';
import { BrowserRouter as Router, Route, Routes } from 'react-router-dom';
import { loadConfig, getConfig } from './utils/config'; // import config here
import Home from './pages/Home';

function App() {
  const [loading, setLoading] = useState(true);

  useEffect(() => {
    const initializeConfig = async () => {
      try {
        await loadConfig();
        setLoading(false);
      } catch (error) {
        console.error('Error loading configuration:', error);
      }
    };

    initializeConfig();
  }, []);

  if (loading) {
    return <div>Loading configuration...</div>;
  }

  return (
    <Router>
      <div>
        <Routes>
          <Route path="/" element={<Home />} />
        </Routes>
      </div>
    </Router>
  );
}

export default App;

Using the configuration inside your application is pretty simple:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

import React from 'react';
import { getConfig } from '../utils/config';

const Example = ({ style }) => {
  const handleExample = () => {
    const appConfig = getConfig();
    console.log(appConfig.settingKey1);
    console.log(appConfig.settingKey2);
    console.log(appConfig.settingKey3);
  };

  return (
    <div style={style}>
      <button onClick={handleExample}>Log Config</button>
    </div>
  );
};

export default Example;

Bonus content: How to deal with configuration in case of Kubernetes

In case you serve your static applications from a simple container with an HTTP server (nginx for example), you can easily mount the configuration file from a ConfigMap.

ConfigMap:

1
2
3
4
5
6
7
8

{{- template "common.configmap" (list . "application_files.configmap") -}}
{{- define "application_files.configmap" -}}
metadata:
  name: {{ template "common.fullname" . }}-files
data:
  config.json: |+
    {{ .Values.configuration | toJson }}  
{{- end -}}

Deployment:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

{{- template "common.deployment" (list . "application.deployment") -}}
{{- define "application.deployment" -}}
spec:
  template:
    metadata:
      annotations:
        checksum/files: {{ include (print $.Template.BasePath "/configmap-files.yaml") . | sha256sum | trunc 63}}
    spec:
      volumes:
        - name: files
          configMap:
            name: {{ template "common.fullname" . }}-files
      containers:
      -
{{ include "common.container.tpl" . | indent 8 }}
        volumeMounts:
          - name: files
            mountPath: /app/config.json
            subPath: config.json
 
{{- end -}}

Do not forget to connect the library chart to render this template correctly:

1
2
3
4
5
6
7
8
9
10

apiVersion: v2
name: example-spa
description: A Helm chart for Example SPA
type: application
version: 1.0.0
appVersion: "1.0.0"
dependencies:
  - name: common
    repository: https://kharkevich.github.io/helm-generic/
    version: 2.1.0

Introduction

Helm is a powerful tool for managing Kubernetes applications. It simplifies the deployment process by using charts, which are packages of pre-configured Kubernetes resources. However, managing multiple charts can become repetitive and error-prone. This is where Library Charts come in handy.

Library Charts on Helm Website

The library chart was introduced at the beginning of Helm 3 (please read the article).

Reasons to Use a Library Chart

Reusability: Share common functionalities and configurations across multiple charts, DRY.

Consistency: All created charts follow the same rules: same labels, same selectors, name conventions.

Maintainability: Majority of the fixes and improvements are centralized. In case you have an independent release flow for your library, all your derivatives will use the defined version, making upgrades straightforward and the impact minimal.

Some of my colleagues still avoid Library Charts and try to create some universal Helm chart for all potential cases/microservices/projects. But over time, all of their “universal” charts turned into chthonic creatures, and support and maintenance became a fight with them.

Anyway, if you have a set of very similar applications, you can create some “half-universal” Helm charts for the set, like:

• backend: ConfigMap (with injection into environment variables), Deployment, Service, HorizontalPodAutoscaler
• single page application: ConfigMap (mount application configuration), Deployment, Service, HorizontalPodAutoscaler

TL;DR

Helm Chart Library you can use from here: helm-generic

1
2
3
4
5
6
7
8
9
10

apiVersion: v2
name: example-chart
description: Example
type: application
version: 1.0.0
appVersion: "1.0.0"
dependencies:
  - name: common
    repository: https://kharkevich.github.io/helm-generic/
    version: 2.1.0

Helm Chart with the Library under the hood: mlflow-tracking-server

Defining a Library Chart

To define the chart as a library, we need to do the following. In Chart.yaml, you need to set type: library like this:

1
2
3
4
5
6
7

apiVersion: v2
name: common
description: Common helm library
type: library
appVersion: 1.0.0
version: 1.0.0
home: https://github.com/kharkevich/helm-generic

To follow Helm concepts, all your helpers in the library chart should begin with an underscore (_) to avoid rendering and output as a Kubernetes manifest file.

So, your templates folder will look like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

_affinity.yaml
_chartref.yaml
_configmap.yaml
_container.yaml
_deployment.yaml
_deployment_strategy.yaml
_envvar.yaml
_fullname.yaml
_healthcheck.yaml
_hpa.yaml
_metadata.yaml
_metadata_annotations.yaml
_metadata_labels.yaml
_name.yaml
_persistentvolumeclaim.yaml
_podmonitor.yaml
_resources.yaml
_secret.yaml
_security_context.yaml
_service.yaml
_serviceaccount.yaml
_servicemonitor.yaml
_tolerations.yaml
_util.yaml
_volume.yaml

Function common.util.merge is one of the key and remarkable functions from the library which allows us to create highly customizable Helm charts with our library. Let’s take a look at the _util.yaml file, where this helper function is located:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

{{- /*
common.util.merge will merge two YAML templates and output the result.

This takes an array of three values:
- the top context
- the template name of the overrides (destination)
- the template name of the base (source)

*/ -}}
{{- define "common.util.merge" -}}
{{- $top := first . -}}
{{- $overrides := fromYaml (include (index . 1) $top) | default (dict ) -}}
{{- $tpl := fromYaml (include (index . 2) $top) | default (dict ) -}}
{{- toYaml (merge $overrides $tpl) -}}
{{- end -}}

To demonstrate how this function works, let’s check how to use it to define a ConfigMap, for example.

1
2
3
4
5
6
7
8
9

{{- define "common.configmap.tpl" -}}
apiVersion: v1
kind: ConfigMap
{{ template "common.metadata" . }}
data: {}
{{- end -}}
{{- define "common.configmap" -}}
{{- template "common.util.merge" (append . "common.configmap.tpl") -}}
{{- end -}}

An empty ConfigMap will be rendered after using this helper in your Helm chart.

1
2
3
4
5
6
7
8
9
10
11
12

apiVersion: v1
data:
kind: ConfigMap
metadata:
  labels:
    app.kubernetes.io/component: api
    app.kubernetes.io/instance: demo-service
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: helm-demo
    app.kubernetes.io/version: "1.0"
    helm.sh/chart: helm-demo-1.0.0
  name: demo-service

To extend this ConfigMap we can do the following:

• set some default required parameters
• add additional code to give us flexibility and customization via values files.

For example

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

{{- template "common.configmap" (list . "application.env_configmap") -}}
{{- define "application.env_configmap" -}}
data:
{{- if .Values.oidc.enabled }}
  SECRET_KEY: {{ required "Required for OIDC configuration" .Values.oidc.secret_key | quote }}
  OIDC_REDIRECT_URI: {{ required "Required for OIDC configuration" .Values.oidc.redirect_uri | quote }}
  OIDC_DISCOVERY_URL: {{ required "Required for OIDC configuration" .Values.oidc.discovery_url | quote }}
  ...
  REDIS_PORT: {{ required "Required for Redis cache" .Values.oidc.redis_port | quote }}
  {{- end }}
  {{- if .Values.oidc.group_detection_plugin }}
  OIDC_GROUP_DETECTION_PLUGIN: {{ .Values.oidc.group_detection_plugin | quote }}
  {{- end }}
{{- end }}
  {{- with .Values.config.data }}
  {{- range $key, $value := . }}
  {{ $key | upper }}: {{ $value | quote }}
  {{- end }}
  {{- end }}
{{- end -}}

Conclusion

Library Charts in Helm provide a powerful way to manage and reuse common configurations across multiple charts. By using Library Charts, you can ensure consistency, improve maintainability, and save time by avoiding repetitive tasks. Explore the helm-generic library to get started with creating your own Library Charts.

How DLP works (simplified)

When I (the user) access a resource from my web client/browser (it can be any application that uses HTTP for communication), the browser connects through corporate DLP. For example, when I try to get content from the https://example.technicaldomain.xyz website, corporate DLP generates a TLS certificate for the website signed by the enterprise internal CA and acts as a proxy. The content and transferred data go through validation and might be blocked.

Yes, this is a real man-in-the-middle attack but enterprise-approved. Otherwise, DLP cannot access the HTTP content under TLS.

Some simplified visual explanations can be found below.

What’s wrong with DLP?

Nothing! DLP is used to protect sensitive data, prevent insider threats, reduce risks of data breaches, give visibility into data usage, and much more. Normally, the enterprise CA is issued and automatically delivered into the system trust store by enterprise administrators. After CA installation, all applications that use the system trust store will be ready to work with DLP without any issues. However, many applications are not ready to work in DLP-enabled environments.

Just a few examples:

• If an application uses certificate pinning, it will be broken in that environment, and you need to adjust DLP rules to allow bypass for all endpoints that utilize certificate pinning for protection.
• All Node.js applications will be broken. npm never looks at the system trust store and has its own CA bundle, which is not editable in a straightforward way.
• All Java-based applications also ignore the system trust store and look for CA in the Java-specific trust store.
• Many Python applications rely on the python-certifi package for getting CA certificates.

Can we fix it? Yes, we can.

All examples below assume that you containerize your application; however, all of these approaches are fair enough for local development environments, application rollout on pure OS, etc. All you need is to adjust the example for your particular case.

Add private CA to your Docker base images

It makes sense to build a special base Docker image with your private CA (and some additional tools for the future) if you work on an application that will be containerized later.

We can create the following:

Dockerfile:

1
2
3

FROM scratch
COPY . /
USER nobody

enterprise-ca.crt certificate file which contains all necessary private CA

1
2
3
4
5

-----BEGIN CERTIFICATE-----
7xFSkyyBNKr79X9DFHOHZF34tHLJRq661V6bY7R+zdRcAitMOeGylZqPHWykYk0x
....
x0kYkyWHPqZlyGeOMtiAcRdz+R7Yb6V166qRJLHt43FZHOHFD9X97rKNByykSFx7
-----END CERTIFICATE-----

jks-add-ca.sh script which will be used to embed our private CA into the Java key store

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

temporary_directory=$( mktemp -d -t i_hate_dlp.XXXXXX )

cleanup() {
    rm -rf ${temporary_directory}
}

trap cleanup EXIT

cd ${temporary_directory}

csplit -s -z $1  '/-----BEGIN CERTIFICATE-----/' '{*}'

STOREPASS="changeit"

counter=0

for certificate in xx*; do
    alias_name="corporate_ca_$counter"
    keytool -import -trustcacerts -cacerts \
    -alias $alias_name \
    -file $certificate \
    -storepass $STOREPASS -noprompt
    ((counter++))
done

Python

A lot of Python-based applications still use the requests package to interact with HTTP resources. To add some extra CA certificates, you just need to define an environment variable with the path to the CA file, like export REQUESTS_CA_BUNDLE=/path/to/your/ca/bundle.crt.

Let’s make a Dockerfile to define it. We will use the Dockerfile with Enterprise CA built in the previous step:

1
2
3
4
5
6
7

ARG CORP_CA_VERSION=1.2.3
ARG PYTHON_VERSION=3.12-alpine
FROM oci.corp.registry/corp-certificates:${CORP_CA_VERSION} AS certificates
FROM python:${PYTHON_VERSION} AS python
COPY --from=certificates /enterprise-ca.crt /usr/local/share/ca-certificates/enterprise-ca.crt
RUN update-ca-certificates
ENV REQUESTS_CA_BUNDLE=/usr/local/share/ca-certificates/enterprise-ca.crt

But some Python packages use their own methods to interact with HTTP and don’t use the requests package. For example, fastapi and httpx.
Here we have the ultimate solution to force our Python to look at the system trust store - truststore. Simple and clever to use.

1
2

import truststore
truststore.inject_into_ssl()

Java

For Java-based applications, we will have almost the same approach. Let’s make a Dockerfile with imported certificates:

1
2
3
4
5
6
7
8
9

ARG CORP_CA_VERSION=1.2.3
ARG JAVA_VERSION=17-alpine3.20
FROM oci.corp.registry/corp-certificates:${CORP_CA_VERSION} AS certificates
FROM amazoncorretto:${JAVA_VERSION} AS java
RUN /bin/sh -c set -eux; apk add --no-cache --allow-untrusted --repository http://dl-cdn.alpinelinux.org/alpine/v3.20/main ca-certificates bash coreutils
COPY --from=certificates /enterprise-ca.crt /usr/local/share/ca-certificates/enterprise-ca.crt
RUN update-ca-certificates
COPY --from=certificates /jks-add-ca.sh /usr/bin/jks-add-ca.sh
RUN bash /usr/bin/jks-add-ca.sh /usr/local/share/ca-certificates/enterprise-ca.crt

Node.js and extra CA

For all of your Node.js applications, you can easily specify the environment variable NODE_EXTRA_CA_CERTS, like this:

1
2
3
4
5
6
7
8

ARG CORP_CA_VERSION=1.2.3
ARG NODE_VERSION=lts-alpine3.20
FROM oci.corp.registry/corp-certificates:${CORP_CA_VERSION} AS certificates
FROM node:${NODE_VERSION} as node
RUN /bin/sh -c set -eux; apk add --no-cache --allow-untrusted --repository http://dl-cdn.alpinelinux.org/alpine/v3.20/main ca-certificates bash coreutils
COPY --from=certificates /enterprise-ca.crt /usr/local/share/ca-certificates/enterprise-ca.crt
RUN update-ca-certificates
ENV NODE_EXTRA_CA_CERTS=/usr/local/share/ca-certificates/enterprise-ca.crt

Certificate Pinning

In case your application uses certificate pinning, you have no choice but to add an exclusion to bypass the route. No workarounds here.

Conclusion

Corporate DLPs and private CA is great stuff, it is used for a lot of enterprises for a good intentions, but makes System Development Life Cycle a little bit challengeable. Fortunately, we know how to deal with it now.