A place to gather bits and bytes.

About

Sat, 06 Apr 2019 21:38:52 +0800

Built with Hugo.

Karlos Nasar in Bahrain 2024

Sat, 28 Dec 2024 21:51:17 +0200

On 11th December 2024, Bulgarian weightlifter Karlos Nasar achieved something extraordinary again by setting two new World Records in the 89kg category: a 183kg Snatch and 405kg Total.

It was amazing!

Android Wireless Pairing Issues

Thu, 27 Jun 2024 07:40:26 +0300

Sometimes Android wireless pairing from Device Manager UI does not work. This may happen because of WIFI network settings, which are not under our control (i.e. co-working space, cafe, hotel).

A fix that may work is using the adb CLI commands for manual pairing.

Go to phone Settings->Developer Options->Wireless Debugging and click Pair device with pairing code.
A dialog shows Pair Code plus device IP address and port.

Note that IP:PORT in this dialog is different from your local device IP:PORT reported on the previous screen. We’ll use the other one later. For pairing we use must use this one in the Dialog window.

Open terminal on development computer and type:

`1`	`adb pair IP:PORT`

Enter Pairing Code displayed on your phone screen. Pairing should succeed.
Now connect to the device by using the other port number displayed before the Pairing dialog popup.

`1`	`adb connect IP:PORT`

Now phone should be paired successfully with the dev computer.

Download Youtube/Facebook Video as MP3 or MP4

Wed, 29 May 2024 16:55:19 +0300

Install or download yt-dlp

Open terminal.

MP3 (Audio)

`1`	`yt-dlp --verbose --extract-audio --audio-format mp3 --audio-quality 0 https://www.youtube.com/watch\?v\=9NbeL4FtcN0`

Audio quality is number from 0 (best) to 10 (worst).

MP4 (Video)

`1`	`yt-dlp --verbose -S "res:720" -f "bestvideo[ext=mp4]+bestaudio[ext=m4a]/best[ext=mp4]/best" https://www.youtube.com/watch\?v\=9NbeL4FtcN0`

Works for Facebook videos too with URL like: https://www.facebook.com/watch/?v=280191971840839

Wrong Golang test coverage in Gitlab CI

Thu, 18 Apr 2024 23:57:52 +0300

Recently I’ve been working on a small Go library.

I wrote unit tests to cover most of the major functionality.

I got code coverage 33%

And I became suspicious.

Why 33%?

Because go test -race -coverprofile=coverage.out ./... includes all auto-generated Fake objects used by unit tests in the calculation. Depending on how many and how large the fake objects are, real code coverage is reduced by that much. After removing fakes from the calculation, code coverage was raised to 62%!

How we measure code coverage in Gitlab CI

unit tests:
  image: golang:1.22.2
  stage: test
  script:
    - go test -race -coverprofile=coverage.out ./...
    - go tool cover -func=coverage.out
  coverage: '/total:\s+\(statements\)\s+(\d+.\d+\%)/'

This will include all Fake objects code and will reduce the real code coverage of our tests.

How it should be

If we want to use go tool cover and our fake files start with the prefix fake_ (i.e. counterfeiter):

unit tests:
  image: golang:1.22.2
  stage: test
  script:
    - go test -race -coverprofile=coverage.out.tmp ./...
    - cat coverage.out.tmp | grep -v "fake_" > coverage.out
    - go tool cover -func=coverage.out
  coverage: '/total:\s+\(statements\)\s+(\d+.\d+\%)/'

Conclusion

If you made the same mistake, you must know that your code coverage is much higher than you, and everybody else think it is :)

Cheers!

Goa Framework Zip File Response

Wed, 08 Nov 2023 23:54:30 +0200

I recently learned an interesting feature in the Goa framework, which we use for some of our services. How can we return a ZIP file which is created by a service function implementation, but also include additional custom headers in the response?

Big thanks to one of the Goa creators/contributors Raphael Simon, who helped me with this issue very quickly in the Goa Slack channel!

Encoding bytes

Goa provides default response encoders for JSON, XML, GOB and plain text/bytes. In my case the plain bytes response was almost enough, but I wanted to have additional response headers returned to the client.

	Method("ExportZip", func() {
		Description("Export a signed zip file.")
		Payload(ExportZipRequest)
		Result(Bytes)
		HTTP(func() {
			GET("/{path}/export")
			Response(StatusOK)
		})
	})

As I wanted to have Content-Type: application/zip header, I wrote a small encoder using the default TextEncoder as example and everything worked fine.

What’s the problem

I wanted to have one additional response header specifying the name of the ZIP file which was dependent on some internal resource characteristics. More specific, I wanted to include a response header Content-Disposition: attachment; filename="myfile.zip" where myfile.zip could be different with every request.

The simple text/bytes encoder cannot do this, because there’s no way I can give it more specific values from the Service implementation - I can only return []byte slice as response.

With Goa, if you need to return more specific headers, you have to define them in a Goa DSL response type.

	var ExportZipResult = Type("ExportZipResult", func() {
		Field(1, "body", Bytes, "Body is the zip file bytes.")
		Field(2, "content-disposition", String, "Content-Disposition response header containing the name of the file.")
		Required("body", "content-disposition")
	})


	Method("ExportZip", func() {
		Description("Export a signed zip file.")
		Payload(ExportZipRequest)
		Result(ExportZipResult)
		HTTP(func() {
			GET("/{path}/export")
			Response(StatusOK, func(){
				Header("content-disposition")				
			})
		})
	})

Now my small custom bytes encoder receives this object, but the body is wrapped in a generated wrapper object which I must cast to the specific ExportZipResult type in order to get the body bytes. It didn’t seem natural as it means that this bytes encoder will only be usable for that specific function and won’t be useful for other similar functions. That’s when I decided to write in the Slack channel and to my surprise the creator of Goa responded within a few hours.

The solution

It turned out that Goa provides a custom DSL function which instructs the code generation process to skip the creation of any response encoder for that function. Instead, you must implement a function which returns an object containing all header values which you’d like to return, plus an io.ReadCloser for the bytes that you want to return to the client.

	var ExportZipResult = Type("ExportZipResult", func() {
		Field(1, "content-type", String, "Content-Type response header.")
		Field(2, "content-disposition", String, "Content-Disposition response header containing the name of the file.")
		Required("content-type", "content-disposition")
	})


	Method("ExportZip", func() {
		Description("Export a signed zip file.")
		Payload(ExportZipRequest)
		Result(ExportZipResult)
		HTTP(func() {
			GET("/{path}/export")

			// bypass response body encoder code generation, so that
			// an io.ReadCloser can be returned to the client
			// while specific response headers can be specified
			// in the ExportZipResult type.
			SkipResponseBodyEncodeDecode()

			Response(StatusOK, func(){
				Header("content-type")
				Header("content-disposition")				
			})
		})
	})

Now there is no need for writing any custom Encoder and the function that implements the creation of the ZIP archive looks like this.

func (s *Service) ExportZip(ctx context.Context, req *mysvc.ExportZipRequest) (*mysvc.ExportZipResult, io.ReadCloser, error) {
	// prepare the ZIP archive []byte slice
	myzip := createSignedZip(...) 

	// prepare custom header value
	contentDisposition := fmt.Sprintf(...)

	return &mysvc.ExportZipResult{
		ContentType: "application/zip",
		ContentDisposition: contentDisposition,
	}, io.NopCloser(bytes.NewReader(myzip)), nil
}

And it works like charm! :)

Insert data in BigQuery JSON Column

Tue, 26 Sep 2023 23:12:13 +0300

Imagine you have a struct with some fields which are always there, but you also have some fields which are dynamically populated with different [key, value] pairs, which cannot be described with static types upfront.

Example Record

type Order struct {
    ID         string                 `json:"id"`
	ItemID     string                 `json:"item_id"`
    Category   string                 `json:"category"`
    Timestamp  int64                  `json:"timestamp"`
    Attributes map[string]interface{} `json:"attributes"`
}

The attributes field is map and may contain an arbitrary number of values for each record. This is the field that I want to store in JSON column. The other fields are stored in statically defined columns with their respective types.

First attempt

BigQuery infer schema does not work with maps. So doing something like this won’t work.

`1`	`schema, err := bigquery.InferSchema(Order{})`

Even if you make the Attributes field type json.RawMessage it won’t work again.

Second attempt

Manually create the schema and try again.

schema := bigquery.Schema{
    {Name: "id", Type: bigquery.StringFieldType},
    {Name: "item_id", Type: bigquery.StringFieldType},
    {Name: "category", Type: bigquery.StringFieldType},
    {Name: "timestamp", Type: bigquery.IntegerFieldType},
    {Name: "attributes", Type: bigquery.JSONFieldType},
}

// ... create the table

err := table.Inserter().Put(ctx, order)

This won’t work (with the Golang client) as BigQuery won’t convert the map to JSON. If we change the field type to json.RawMessage it won’t work again.

Third attempt is OK

We make the Order type implement the bigquery.ValueSaver interface and serialize the attributes field as JSON string.

type OrderValueSaver struct {
    order *Order
}

// Save implements the bigquery.ValueSaver interface 
// to serialize the "attributes" field as JSON string.
func (s *OrderValueSaver) Save() (row map[string]bigquery.Value, insertID string, err error) {
    attr, err := json.Marshal(s.order.Attributes)
    if err != nil {
        return nil, "", err
    }

    row = map[string]bigquery.Value{
        "id":         s.order.ID,
        "item_id":    s.order.ItemID,
        "category":   s.order.Category,
		"timestamp":  s.order.Timestamp,
        "attributes": string(attr),
    }

    return row, s.order.ID, nil
}

func (c *Client) SaveOrder(ctx context.Context, order *Order) error {
    schema := bigquery.Schema{
        {Name: "id", Type: bigquery.StringFieldType},
        {Name: "item_id", Type: bigquery.StringFieldType},
        {Name: "category", Type: bigquery.StringFieldType},
        {Name: "timestamp", Type: bigquery.IntegerFieldType},
        {Name: "attributes", Type: bigquery.JSONFieldType},
    }
	
	// ... create/get table, ideally do it in the constructor of the client once 
	
	saver := &OrderValueSaver{order: order}
    
	return table.Inserter().Put(ctx, saver)
}

Ebiten Android and Go Callbacks

Mon, 10 Jul 2023 22:19:20 +0300

Ebitengine is a Go game development engine. Games can be built for Linux, MacOS, Windows, Android, iOS and Nintendo Switch.

I did some basic experiments during a few vacation days. This post is about something which wasn’t obvious to me how to implement and took me some hours to figure out. Hopefully it may spare you some time.

Under the hood Ebitengine uses gomobile to create bindings for languages like Java and C/C++.

This article is especially interesting if you need to call Go functions from Java or C/C++ and vice-versa.

What is the problem

I make a Go game which runs on Android, but I’d like to show Ads using 3rd party SDK like Unity or Facebook Audience Network. This means I’ll have to integrate a Java or ObjC library in the native app. For example, on Android it could be implemented in the showAds method below:


import go.Seq;
import com.mycompany.mygame.mobile.EbitenView;

public class MainActivity extends AppCompatActivity {
    protected EbitenView gameView;

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        Seq.setContext(getApplicationContext());

        gameView = new EbitenView(getApplicationContext());
        setContentView(gameView);
    }

    @Override
    protected void onPause() {
        super.onPause();
        gameView.suspendGame();
    }

    @Override
    protected void onResume() {
        super.onResume();
        gameView.resumeGame();
    }

    public void showAds() {
    	...
    }
}

The problem is that MainActivity doesn’t know when to call show ads, as it doesn’t know anything about the internal state of the game - game updates and state changes happen in the Go code. So, at any point in time, only the Go side knows if it’s time to show an advert. Now the problem comes down to how the Go code can call the showAds() Java method?

Solution

My current solution is presented below and it works. I don’t feel it optimal since it requires to keep a global variable reference to the game object, but for now I haven’t found an improvement (and I don’t know if it’s an issue at all).

It shows how to call Go functions from Java as well as Java functions from Go.

Go

First, we must tell the Go side what function it should call to show ads, as it doesn’t know anything about the existence of the MainActivity.showAds method. This means I must call a Go function from Java to register the callback.

To do that I define an exported Go function in the mobile.go file which will be used to produce the bindings. The function will be exported as static class method on the Java side.

I also define an interface for the object that I expect to receive as an argument. The object must implement a method called showAds and will be implemented on the Java side. The exported function RegisterAdsCallback is used from Java to give this object to the Go side.

package mobile

import (
    ...
)

var g *game.Game

func init() {
    var err error
    g, err = game.New()
    if err != nil {
        panic(err)
    }
    mobile.SetGame(g)
}

type AdsCallback interface {
    ShowAds()
}

func RegisterAdsCallback(callback AdsCallback) {
    g.RegisterAdsCallback(func() { callback.ShowAds() })
}

Now when I produce the Java library containing the game with ebitenmobile, the class Mobile will have this RegisterAdsCallback static method, which can be used from Java to register the showAds method.

`1`	`ebitenmobile bind -target android -androidapi 19 -javapkg com.mycompany.mygame -o /path/to/androidapp/libs/mygame.aar /path/to/mobile.go`

Java

In Java we implement the callback and give it to the Go side.

import com.mycompany.mygame.mobile.AdsCallback;
import com.mycompany.mygame.mobile.EbitenView;
import com.mycompany.mygame.mobile.Mobile;

public class MainActivity extends AppCompatActivity {
    EbitenView gameView = null;

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        Seq.setContext(getApplicationContext());

        Mobile.registerAdsCallback(new AdsCallback() {
            @Override
            public void showAds() {
                Log.d("mygame", "show ads");
            }
        });

        gameView = new EbitenView(getApplicationContext());
        setContentView(gameView);
    }

    ...
}

Usage in Go

Whenever the Go game decides that it’s time to show ads, it can invoke game.showAds():

package game

type Game struct {
    adsCallback func()
}

func New() (*Game, error) {
    return &Game{}, nil
}

func (g *Game) Update() error {
    ...
}

func (g *Game) Draw(screen *ebiten.Image) {
    ...
}

func (g *Game) Layout(outsideWidth, outsideHeight int) (int, int) {
    ...
}

func (g *Game) RegisterAdsCallback(callback func()) {
    g.adsCallback = callback
}

func (g *Game) showAds() {
    g.adsCallback()
}

Convert Aries KMS public key to JWK

Wed, 29 Mar 2023 10:37:41 +0300

It looks like neither Hyperledger Aries Go framework, nor the Google Tink crypto library, which it uses under the hood, provide helper functions to convert public key bytes to JWK format (or I haven’t found the way). When you export public key bytes from the KSM, they are unmarshalled from their internal representation and serialized using x509.MarshalPKIXPublicKey(), elliptic.Marshal(), and some are marshalled as JSON using json.Marshal() to custom Aries PublicKey struct. This means that we can’t use standard Go JOSE libraries to convert these public keys to JWK directly.

This post shows how to make two kinds of keys to JWK, but I’ll try to explain the process I followed to discover them, so if you need to export other public key types, you’ll probably know where to look.

One of the key types - NISTP256ECDHKWType - is mandatory in the current implementation of Hyperledger Aries Go if you want to use authenticated encryption (authcrypt) of DIDComm messages and follow the DIDComm V2 spec.

Why would you want to convert Aries public keys to JWK?

To use them as verification methods in DID documents. It is a common and widely supported encoding of public keys in DID documents (publicKeyJwk)

Below are example functions to convert ECDSA and ECDSA-ECDH-KW keys:

type JWK struct {
	Kty string `json:"kty"`
	Crv string `json:"crv"`
	X   string `json:"x"`
	Y   string `json:"y"`
	Kid string `json:"kid"`
}

func main() {
	// Aries KMS is created and initialized and you have a reference to it.
	// ...

	keyID := "..."
	pubkey, keyType, err := ariesKMS.ExportPublicKeyBytes(keyID)
	if err != nil {
		log.Fatalln(err)
	}

	jwk, err := PublicKeyToJWK(keyID, keyType, pubKey)
	if err != nil {
		log.Fatalln(err)
	}

	jwkBytes, err := json.Marshal(jwk)
	if err != nil {
		log.Fatalln(err)
	}

	log.Printf("jwkBytes = %#+v\n", jwkBytes)
}


// PublicKeyToJWK converts public key bytes from the binary format of the Tink
// key to a JWK.
func PublicKeyToJWK(keyID string, kt kms.KeyType, pubKey []byte) (interface{}, error) {
	switch kt {
	case kms.ECDSAP256TypeIEEEP1363, kms.ECDSAP384TypeIEEEP1363, kms.ECDSAP521TypeIEEEP1363:
		return ecKeyToJwk(keyID, kt, pubKey)
	case kms.NISTP256ECDHKWType, kms.NISTP384ECDHKWType, kms.NISTP521ECDHKWType:
		return ecdhKWToJwk(keyID, pubKey)
	default:
		return nil, fmt.Errorf("unsupported key type: %v", kt)
	}
}

func ecKeyToJwk(keyID string, kt kms.KeyType, pubKey []byte) (interface{}, error) {
	var x, y *big.Int
	var curve elliptic.Curve

	switch kt {
	case kms.ECDSAP256TypeIEEEP1363:
		curve = elliptic.P256()
		x, y = elliptic.Unmarshal(curve, pubKey)
	case kms.ECDSAP384TypeIEEEP1363:
		curve = elliptic.P384()
		x, y = elliptic.Unmarshal(curve, pubKey)
	case kms.ECDSAP521TypeIEEEP1363:
		curve = elliptic.P521()
		x, y = elliptic.Unmarshal(curve, pubKey)
	default:
		return nil, fmt.Errorf("unsupported key type: %v", kt)
	}

	xEncoded := base64.RawURLEncoding.EncodeToString(x.Bytes())
	yEncoded := base64.RawURLEncoding.EncodeToString(y.Bytes())

	jwk := JWK{
		Kty: "EC",
		Crv: curve.Params().Name,
		X:   xEncoded,
		Y:   yEncoded,
		Kid: keyID,
	}

	return jwk, nil
}

func ecdhKWToJwk(keyID string, pubKey []byte) (interface{}, error) {
	var ariesKey cryptoapi.PublicKey
	if err := json.Unmarshal(pubKey, &ariesKey); err != nil {
		return nil, err
	}

	xEncoded := base64.RawURLEncoding.EncodeToString(ariesKey.X)
	yEncoded := base64.RawURLEncoding.EncodeToString(ariesKey.Y)

	jwk := JWK{
		Kty: ariesKey.Type,
		Crv: altCurveName(ariesKey.Curve),
		X:   xEncoded,
		Y:   yEncoded,
		Kid: keyID,
	}

	return jwk, nil
}

func altCurveName(name string) string {
	switch name {
	case "NIST_P256":
		return "P-256"
	case "NIST_P384":
		return "P-384"
	case "NIST_P521":
		return "P-521"
	default:
		return name
	}
}

How to find serializations for other supported key types?

By looking at the public key serialization process for your needed key type.

Keys are created by different KeyManager implementations.

Public keys are serialized in the pubkey_writer.go file. Best to checkout the Hyperledger Aries Go github repo and dive into this file and follow the different serialization functions for the key types you need.

go-watcher replacement for Go 1.20+

Thu, 02 Mar 2023 15:35:32 +0200

We’ve been using go-watcher for local development with docker-compose to automatically rebuild a given service when its source code changes. From Go 1.20+ onwards the go-watcher binary no longer works, because it’s using a build flag -i which has been deprecated for a long time, but now is finally removed. Because of that the compilation fails with error. See it here.

The project is no longer maintained, and it seems that the authors are not going to update it.

Now I’m using a replacement called guard. Usage is almost identical as you can see in the examples below.

Note: we commit vendor, that’s why we build/install with -mod=vendor

Example Dockerfile using go-watcher to watch and rebuild a service.

FROM golang:1.19.5

RUN go install github.com/canthefason/go-watcher/cmd/watcher@v0.2.4

ADD . /go/src/gitlab.com/myuser/myproject

WORKDIR /go/src/gitlab.com/myuser/myproject

RUN go install -mod=vendor ./cmd/svcmain/...

EXPOSE 8080

ENTRYPOINT ["sh", "-c", "/go/bin/watcher -run gitlab.com/myuser/myproject/cmd/svcmain -watch gitlab.com/myuser/myproject"]

Example Dockerfile using guard to watch and rebuild a service.

FROM golang:1.20.1

RUN go install github.com/ysmood/kit/cmd/guard@v0.25.11

ADD . /go/src/gitlab.com/myuser/myproject

WORKDIR /go/src/gitlab.com/myuser/myproject

EXPOSE 8080

ENTRYPOINT ["sh", "-c", "/go/bin/guard -w '**/*.go' -- go run -mod=vendor ./cmd/svcmain/..."]

Easy writing of git tag messages

Fri, 25 Nov 2022 23:23:06 +0200

Display all commit messages since the last tag:

`1`	`git log --no-merges --oneline --format=%s $(git describe --tags --abbrev=0)..HEAD`

Vault Dev Server in Docker Compose

Mon, 03 Oct 2022 23:20:14 +0300

I have difficulties using the Vault dev server mode in docker-compose environment. Other people have difficulties too.

Partly it’s due to incomplete/imperfect Vault documentation. I had to skim through the source code to understand what parameters can be used for some commands and for the dev server itself. Some of these were hard to find in the documentation.

Objective

I want Vault running in compose env with these features enabled on start:

automatically unsealed
enabled Vault Transit Engine
two signing keys in Transit Engine, one ECDSA and one ED25519 with names key1 and key2
predefined root token which can be embedded in all services needing to communicate with it

Here is a solution that works well.

TL;DR

Just use this. If you want to read some details follow below.

docker-compose.yml

  vault:
    hostname: vault
    container_name: vault
    image: vault:1.10.3
    environment:
      VAULT_ADDR: "http://0.0.0.0:8200"
      VAULT_API_ADDR: "http://0.0.0.0:8200"
    ports:
      - "8200:8200"
    volumes:
      - ./volumes/vault/file:/vault/file:rw
    cap_add:
      - IPC_LOCK
    entrypoint: vault server -dev -dev-listen-address="0.0.0.0:8200" -dev-root-token-id="root"

  vault-init:
    container_name: vault-init
    image: vault:1.10.3
    volumes:
      - ./vault-init.sh:/vault-init.sh
    depends_on:
      - vault
    restart: "no"
    entrypoint: sh -c "/vault-init.sh"

vault-init.sh

#! /bin/sh

set -e

export VAULT_ADDR=http://vault:8200

# give some time for Vault to start and be ready
sleep 3

# login with root token at $VAULT_ADDR
vault login root

# enable vault transit engine
vault secrets enable transit

# create key1 with type ed25519
vault write -f transit/keys/key1 type=ed25519

# create key2 with type ecdsa-p256
vault write -f transit/keys/key2 type=ecdsa-p256

That’s it.

Details

The dev server is always unsealed by default, so our first objective is achieved.

The flag -dev-listen-address allows to bind to all interfaces, otherwise the Dev Server automatically binds to 127.0.0.1. This prevents it from being visible from other containers in the docker-compose network. It also prevents the Vault UI from being accessible from the host machine at http://localhost:8200/ui/vault

The flag -dev-root-token-id sets a predefined root token for the dev server. The token can be set as a configuration variable in all services which need to communicate with the Vault. (two objectives achieved)

The other two objectives - enable Transit Engine and create two keys inside it - are achieved by the vault-init.sh script which is started once as a separate container. It connects to the running Vault container at http://vault:8200, enables the Transit Engine and creates two keys.

Most parameters of the command to create keys are also undocumented. For example, to create a key of specific type, we can pass in a key=value pair. Looking at the source code you can see how parameters are read from stdin. To learn what parameters could be used in this (and other commands) I was observing the HTTP requests passed from the Vault UI in the browser to the Vault backend. For example, to create a a key with specific type you can use:

`1`	`vault write -f transit/keys/key1 type=ed25519`

The type parameter is undocumented, but there are many others that can be passed as well, but you’ll have to look through the source code to see what can be used for your specific use case.

Convert Vault Public Key to JWK

Sun, 26 Jun 2022 17:19:20 +0300

Hashicorp Vault Transit Engine API allows signing of data and verification of signatures. It offers more features like encryption/decryption, hashing, random bytes generation, but in this post we’re concerned only with the signing part of the API and more specifically with converting public keys returned by the Vault API to JWK format.

Example response when fetching a public key from Vault:

curl -X GET https://myvault.com/v1/transit/keys/keyname

{
    "data":
    {
        "keys":
        {
            "1":
            {
                "creation_time": "2022-06-13T12:21:27.257274629Z",
                "name": "P-256",
                "public_key": "-----BEGIN PUBLIC KEY-----\nMFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAERTx/2cyYcGVSIRP/826S32BiZxSg\nnzyXgRYmKP8N2l26ec/MwCdsHIEyraX1ZYqwMUT4wO9fqFiGsRKyMBpPnQ==\n-----END PUBLIC KEY-----\n"
            }
        },
        "latest_version": 1,
        "type": "ecdsa-p256"
    }
}

Convert public key to JWK

Some keys like ED25519 are just random bytes and are returned as base64.StdEncoding string, while other key types like ECDSA and RSA have some structure and are returned as PEM encoded keys.

Below is an example function that creates a jose.JSONWebKey from a Vault public key.

package example

import (
    "crypto/ed25519"
    "crypto/x509"
    "encoding/pem"
    "fmt"

    "github.com/square/go-jose/v3"
)	

func jwkFromPEM(keyname string, keytype string, pemKey string) (*jose.JSONWebKey, error) {
    var key interface{}
	
    switch keytype {
    case "ed25519":
        pk, err := base64.StdEncoding.DecodeString(pemKey)
        if err != nil {
            return nil, fmt.Errorf("failed to decode ed25519 key: %v", err)
        }
        key = ed25519.PublicKey(pk)
    case "ecdsa-p256", "ecdsa-p384", "ecdsa-p521", "rsa-2048", "rsa-4096":
        block, _ := pem.Decode([]byte(pemKey))
        if block == nil {
            return nil, fmt.Errorf("no public key found during PEM decode")
        }
		
        var err error
        key, err = x509.ParsePKIXPublicKey(block.Bytes)
        if err != nil {
            return nil, err
        }
    default:
        return nil, fmt.Errorf("unsupported key type: %s", keytype)
    }
	
    return &jose.JSONWebKey{
        KeyID: keyname,
        Key: key,
    }, nil
}

Example JSON encoding of a converted JWK ECDSA-P256 public key would look like:

{
    "kid": "keyname",
    "kty": "EC",
    "crv": "P-256",
    "x": "RTx_2cyYcGVSIRP_826S32BiZxSgnzyXgRYmKP8N2l0",
    "y": "unnPzMAnbByBMq2l9WWKsDFE-MDvX6hYhrESsjAaT50"
}

git reflog

Tue, 14 Jun 2022 00:13:05 +0300

git reflog is underrated.

Gitlab Subgroups and Go Modules Names

Tue, 08 Mar 2022 07:23:31 +0200

I sometimes see people who use private Gitlab subgroups to name their Go
modules (mostly libraries) with a suffix .git

Let’s see an example. If we have a Go module project at the following imaginary location:

`1`	`https://gitlab.com/myuser/foogroup/barlib`

Now when running go get gitlab.com/myuser/foogroup/barlib from another Go project, even though the SSH keys are set correctly, we may get an error:

`1`	`The project you were looking for could not be found or you don't have permissions to view it.`

We start googling and reading stackoverflow and it seems that a possible solution is to rename the module to gitlab.com/myuser/foogroup/barlib.git … and the issue looks fixed.

rm go.mod go.sum
go mod init gitlab.com/myuser/foogroup/barlib.git
go mod tidy

cd /go/src/another_project
go get gitlab.com/myuser/foogroup/barlib.git

Also at this point people start to doubt the goodness of the Go tools (go mod and go get especially) and mark this as a stupid annoying (Go) bug.

The solution is simple and the reason for the issue too and it’s not a bug.

The reason behind that permissions error

Why this happens is that go get tries to discover the modules at a given path in order to find the requested Go module repository. Only after the repository is found, the tools will do git clone or git checkout and the SSH keys will be used for authentication. The issue comes down to the fact that private Gitlab subgroups cannot be listed/viewed without a Gitlab Access Token.

The fix is Gitlab Access Token

Go to Gitlab->Preferences->Access Tokens and create one. Then create (or edit) a .netrc file in your home folder: ~/.netrc The contents of the file are below:

1
2
3

machine gitlab.com
    login my_gitlab_username
    password my_gitlab_token

And we may not need to name our Go modules with a .git suffix anymore.

Update RETURNING with Gorm

Wed, 19 Jan 2022 14:23:32 +0200

In some situations it’s not obvious how to use the RETURNING SQL clause with Gorm and databases that support it, like CockroachDB.

The particular use case where I found it challenging is the following: I want to update multiple records with a WHERE clause, ORDER BY clause and LIMIT clause and immediately return the updated records without a second query.

The simplified CockroachDB table schema without indices and noise is:

  column_name | data_type | is_nullable | column_default  
--------------+-----------+-------------+----------------
  field1      | INT8      |    true     | NULL                                  
  field2      | INT8      |    true     | NULL                                  
  created_at  | INT8      |    true     | NULL

My initial attempts were as follows.

type Record struct {
	Field1    int
	Field2    int
	CreatedAt int64
}         

func update(field1 int, field2 int, limit int) ([]Record, error) {
    var records []Record
    err := db.Model(&records).
    	Table("table_name").
        Clauses(clause.Returning{}).
        Where("field1 = ?", field1).
    	Updates(map[string]interface{}{"field2":field2}).
    	Order("created_at ASC").
    	Limit(limit).
    	Error
    if err != nil {
        return nil, err
    }
    
    return records, nil
}

Gorm builds the SQL statement successfully, but for some reason doesn’t include the ORDER BY and LIMIT clauses, which are very important in my particular case. Ok, let’s try with a raw SQL statement.

func update(field1 int, field2 int, limit int) ([]Record, error) {
    var records []Record
    sql := `UPDATE table_name SET field2 = $1 WHERE field1 = $2 ORDER BY created_at ASC LIMIT $3 RETURNING *`
    if err := db.Model(&records).Table("table_name").Exec(sql, field1, field2, limit).Error; err != nil {
        return nil, err
    }
    return records, nil
}

Gorm executes the complete correct query, but I couldn’t find a way to return/populate the records slice - it was always empty (tried with different arrangements and examples from the docs). Without any deep investigation on why that happens, it seems that Gorm just ignores the RETURNING data when executing raw SQL statements.

What finally worked for me seems a bit like a hack, but it does the job. By using the ORDER BY and LIMIT clauses directly in the WHERE clause, the query works correctly as I need it.

func update(field1 int, field2 int, limit int) ([]Record, error) {
    var records []Record
    err := db.Model(&records).
    	Table("table_name").
    	Clauses(clause.Returning{}).
    	Where("field1 = $1 ORDER BY created_at LIMIT $2", field1, limit).
    	Updates(map[string]interface{}{"field2": field2}).
    	Error
    if err != nil {
        return nil, err
    }
    return records, nil

The examples use Updates with map[string]interface{} but the issue described here is not affected by that. Using struct or other ways for updating produces the same results.

note

It looks like PostgreSQL does not work with LIMIT when using a RETURNING clause so another solution could be used, like SELECT FOR UPDATE -> then update, and return the updated record.

Graceful Shutdown for Go HTTP servers

Sun, 25 Apr 2021 19:12:56 +0300

A simple Go implementation for graceful shutdown of HTTP servers with Go 1.8+ wrapped in ~20 lines of code.

func main() {
	srv := &http.Server{
		Addr:    ":8080",
		Handler: http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
			time.Sleep(5 * time.Second)
			_, _ = w.Write([]byte("hello"))
		}),
	}

	if err := gracefulShutdown(srv, 10*time.Second); err != nil {
		log.Println(err)
	}
}

// gracefulShutdown stops the given HTTP server on
// receiving a stop signal and waits for the active connections
// to be closed for {timeout} period of time.
func gracefulShutdown(srv *http.Server, timeout time.Duration) error {
	done := make(chan error, 1)
	go func() {
		c := make(chan os.Signal, 1)
		signal.Notify(c, os.Interrupt, syscall.SIGTERM)
		<-c

		ctx := context.Background()
		var cancel context.CancelFunc
		if timeout > 0 {
			ctx, cancel = context.WithTimeout(ctx, timeout)
			defer cancel()
		}

		done <- srv.Shutdown(ctx)
	}()

	if err := srv.ListenAndServe(); err != nil && err != http.ErrServerClosed {
		return err
	}

	return <-done
}

The graceful shutdown function can also be used directly as a package from github.com: Graceful Shutdown

info

You should use Go 1.8+

The function doesn’t support TLS servers currently as I don’t need it, but it should be trivial to add that.

Simple GitLab CI for Go

Sat, 13 Mar 2021 00:04:37 +0200

Example for running unit tests with coverage output

image: golang:1.16

stages:
  - test

before_script:
  - ln -s /builds /go/src/gitlab.com
  - cd /go/src/gitlab.com/${CI_PROJECT_PATH}

unit tests:
  stage: test
  script:
    - go version
    - go test -race $(go list ./... | grep -v /vendor/) -coverprofile=coverage.out
    - go tool cover -func=coverage.out

To Vendor or Not to Vendor

Thu, 11 Mar 2021 08:45:04 +0200

Should we commit `vendor` in our Go projects?

Since the introduction of Go modules, the prevalent opinion on the internet is that we should not commit the vendor directory of our projects.

The argument against committing vendor is that it’s unnecessary, because the Go tools can resolve and find dependencies when needed on-the-fly. All public Go modules are also hosted by various Go Proxy servers for redundancy.

While I agree that it’s technically unnecessary and bloats the repo, I’d like to give some arguments in favour of committing vendor.

Building the project doesn’t depend on some code being available on Github/Gitlab/… or the Go Proxy servers. Open source projects may disappear because of censorship, authors incentives, licensing changes or some other reasons I can’t currently think of. Not in your repo, not your code.
We may use internal or 3rd party Go modules (private) which may also disappear or become inaccessible, but if they are committed in vendor, they are part of our project and nothing breaks unexpectedly.
Go modules may not follow semantic versioning, which means the Go tools will rely on the latest commit hash when fetching them on-the-fly. Repo history may be rewritten (e.g. rebase) and you, a colleague or your CI job may end up with different code for the dependencies they use.
CI/CD jobs which perform compilation and build steps need not waste time and network to download the dependencies every time the CI job is executed. All needed dependencies are local and present (go build -mod vendor)
CI/CD jobs which perform compilation and build steps may not be configured additionally to authenticate against private repos, so that they can download the dependencies.
Committing vendor can sometimes improve the code review process. Typically we’re committing dependency changes in a separate commit, so they can be easily viewed if the reviewer is curious.

Here’s an interesting observation related to bloating the repo. If I make code review, and a team member has included a new dependency with 300 files (or updated one with 300 files changed), I may be curious to deep dive into that and start a discussion about the code quality, the need for this change or alternative Go modules. This may lead us to actually decrease our binary size and overall complexity.

If I just see a single new line in go.mod in new MR, chances are I won’t even think about it.

As a pseudo argument I can ask why important open source Go projects like Kubernetes are committing vendor, if it’s not necessary?

Conclusion

For my own peace of mind I would always prefer to commit the vendor directory alongside go.mod and go.sum while at the same time try to keep the project dependencies at minimum.

warning

The vendor directory SHOULD NOT be committed for Go libraries, frameworks or code which doesn’t produce executable binaries, unless you have a really good reason to do it.

Bash script preserve command-line arguments

Thu, 01 Oct 2020 12:06:14 +0300

If we want to preserve (pass) the command line arguments when running a bash script and pass them to a program defined in the script, we can use the $* pattern after the call. An example bash script called myscript.sh is shown below. It sets some environment variables (optional) and starts a program called myserver. The program will accept all command line args passed to the bash script as if they were passed to the myserver program directly.

#!/bin/bash

SOME_ENV_VARIABLE=var1 \
ANOTHER_ENV_VARIABLE=var2 \
myserver $*

Calling the script like:

`1`	`./myscript.sh -addr http://example.com -port 8080`

will be like calling the myserver program like:

`1`	`./myserver -addr http://example.com -port 8080`

A place to gather bits and bytes.

About

Karlos Nasar in Bahrain 2024

Android Wireless Pairing Issues

Download Youtube/Facebook Video as MP3 or MP4

MP3 (Audio)

MP4 (Video)

Wrong Golang test coverage in Gitlab CI

Why 33%?

How we measure code coverage in Gitlab CI

How it should be

Conclusion

Goa Framework Zip File Response

Encoding bytes

What’s the problem

The solution

Insert data in BigQuery JSON Column

Example Record

First attempt

Second attempt

Third attempt is OK

Ebiten Android and Go Callbacks

What is the problem

Solution

Go

Java

Usage in Go

Convert Aries KMS public key to JWK

How to find serializations for other supported key types?

go-watcher replacement for Go 1.20+

Easy writing of git tag messages

Vault Dev Server in Docker Compose

Objective

TL;DR

Details

Convert Vault Public Key to JWK

Convert public key to JWK

git reflog

Gitlab Subgroups and Go Modules Names

The reason behind that permissions error

The fix is Gitlab Access Token

Update RETURNING with Gorm

Graceful Shutdown for Go HTTP servers

Simple GitLab CI for Go

Example for running unit tests with coverage output

To Vendor or Not to Vendor

Should we commit vendor in our Go projects?

Conclusion

Bash script preserve command-line arguments

Should we commit `vendor` in our Go projects?