Friday, 25 September 2020

GNU Core Utility: uname

One of GNU core utility commands is uname. It is a handly tool that gives some useful information about the system.

GNU core utils
Image credit: maizure.org


Manual:

uname
UNAME(1)                                                             User Commands                                                             UNAME(1)
NAME
       uname - print system information
SYNOPSIS
       uname [OPTION]...
DESCRIPTION
       Print certain system information.  With no OPTION, same as -s.
       -a, --all
              print all information, in the following order, except omit -p and -i if unknown:
       -s, --kernel-name
              print the kernel name
       -n, --nodename
              print the network node hostname
       -r, --kernel-release
              print the kernel release
       -v, --kernel-version
              print the kernel version
       -m, --machine
              print the machine hardware name
       -p, --processor
              print the processor type (non-portable)
       -i, --hardware-platform
              print the hardware platform (non-portable)
       -o, --operating-system
              print the operating system
       --help display this help and exit
       --version
              output version information and exit
AUTHOR
       Written by David MacKenzie.
REPORTING BUGS
       GNU coreutils online help: <http://www.gnu.org/software/coreutils/>
       Report uname translation bugs to <http://translationproject.org/team/>
COPYRIGHT
       Copyright © 2017 Free Software Foundation, Inc.  License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
       This is free software: you are free to change and redistribute it.  There is NO WARRANTY, to the extent permitted by law.
SEE ALSO
       arch(1), uname(2)
       Full documentation at: <http://www.gnu.org/software/coreutils/uname>
       or available locally via: info '(coreutils) uname invocation'


To get the name of the kernel:

$ uname -s
Linux

To verify if you're running a 64-bit system use uname tool:

$ uname -m 

Output:
  • 64-bit Intel/AMD system: x86_64
  • 64-bit ARM architecture: aarch64

If you come across instructions about how to install docker-compose on Linux, you'll see an example how to use uname in a command which downloads a binary built for your system:

sudo curl -L "https://github.com/docker/compose/releases/download/1.27.4/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose


Saturday, 12 September 2020

Introduction to Dockerfile


Docker is a layered filesystem so every ADD, COPY and RUN instruction will create a new layer and cache it.


ADD

  • takes in a src and destination
  • lets you copying into the Docker image files/directories from following sources:
    • local file or directory from your host (the machine building the Docker image)
    • you can extract a local tar file from the source directly into the destination
    • URL
  • valid use case for ADD is when you want to extract a local tar file into a specific directory in your Docker image

ARG


  • defines a variable that users can pass at build-time to the builder with the docker build command using the --build-arg <varname>=<value> flag
  • Docker build will always show you the line as is written down in the Dockerfile, despite the variable value - ARG value will not be substituted in the terminal output. [ARG substitution in RUN command not working for Dockerfile]
  • Value of the argument provided in docker build command line will overwrite the (default) one set in Dockerfile.
  • WARNING: all ARG values that are defined before FROM will be reset (empty) after FROM [ARG before FROM in Dockerfile doesn't behave as expected · Issue #34129 · moby/moby]
  • ARG variables are available in build time (RUN, COPY etc...). They are not embedded into image (like ENVs) and therefore can't be used in CMD or ENTRYPOINT commands (like ENVs).

Example:

Dockerfile:


ARG NPM_LOG_LEVEL=warn
RUN npm install --loglevel ${NPM_LOG_LEVEL}


Terminal:

$ docker build --pull --build-arg NPM_LOG_LEVEL=verbose -t my_app_image .


To pass ARG values into container we need to use ENV variables:

ARG APP_NAME=mysqlsh-demo
ARG DOCKER_ENTRYPOINT=docker-entrypoint.sh
...
# ARGs are available only in build time but not runtime so we need to pass their values to ENVs:
ENV APP_NAME=${APP_NAME}
ENV DOCKER_ENTRYPOINT=${DOCKER_ENTRYPOINT}

ENTRYPOINT "/usr/src/${APP_NAME}/${DOCKER_ENTRYPOINT}"


COPY


NOTE for ADD & COPY:
  • All new files and directories are created with a UID and GID of 0, unless the optional --chown flag specifies a given username, groupname, or UID/GID combination to request specific ownership of the content added.
  • Examples:
ADD --chown=someuser:somegroup /foo /bar 
COPY --chown=someuser:somegroup /foo /bar 
Or other combinations of user/group name (or ID); 
--chown=someuser:123 
--chown=anyuser:anygroup 
--chown=1001:1002 
--chown=333:agroupname


CMD

  • Lets you define a default command to run when your container starts
  • Executed in run-time;  does not execute anything at build time
  • Sets default command and/or parameters, which can be overwritten from command line when docker container runs
  • Has three forms:
    • Exec (preferred): CMD ["executable","param1","param2"]
    • Shell: CMD command param1 param2
    • ENTRYPOINT's default parameters list (no binary!): CMD ["param1","param2"]
  • Exec form executes stated executable and passes to it params listed.
  • Shell form invokes a command shell (e.g. sh -c) and passes both command (executable) and its params to it.
  • When used in the shell or exec formats, the CMD instruction sets the command to be executed when running the image.
  • If you would like your container to run the same executable every time, then you should consider using ENTRYPOINT in combination with CMD.
  • If the user specifies arguments to docker run then they will override the default specified in CMD.


ENTRYPOINT

  • Has two forms:
    • Exec (preferred): ENTRYPOINT ["executable", "param1", "param2"]
    • Shell: ENTRYPOINT command param1 param2
  • Exec form executes stated executable and passes to it params listed.
  • Shell form invokes a command shell (e.g. sh -c) and passes both command (executable) and its params to it.
  • Configures a container that will run as an executable. It should be used if container is intended to run the same executable every time. This means that we can pass arguments to the executable set as an entrypoint simply by listing them after the name of the container:          $ docker run ...<container_name> param1 param2...
  • Default values of arguments can be specified with CMD instruction in JSON array format: 
    • CMD ["param1", "param2"]
  • Executed in run-time
  • From Docker best practices:
The best use for ENTRYPOINT is to set the image’s main command, allowing that image to be run as though it was that command (and then use CMD as the default flags). 
Let’s start with an example of an image named same as the binary (s3cmd) for the command line tool s3cmd: 
   ENTRYPOINT ["s3cmd"]
   CMD ["--help"] 
Now the image can be run like this to show the command’s help: 
   $ docker run s3cmd
Or using the right parameters to execute a command: 
   $ docker run s3cmd ls s3://mybucket 
This is useful because the image name can double as a reference to the binary as shown in the command above.

If we named image as my-s3cmd-image we'd run the container as:

# These arguments get passed to s3cmd
docker run my-s3cmd-image ls s3://mybucket
docker run my-s3cmd-image sync s3://source s3://dest
docker run my-s3cmd-image --help
 
To override the entrypoint (if you need to), we'd use:

docker run --entrypoint /bin/bash my-s3cmd-image

We always need the image name in docker run. The ENTRYPOINT just pre-configures what command runs inside that image when it starts.

Nice example of entrypoint.sh.

This is an example how can operator (person who is running container from an image) pass arguments to the executable run upon the container's launch:

Dockerfile:

...
ENTRYPOINT [ "/my-app" ]
CMD [ "--param1=arg1_default" ]

Launching the container:

$ docker run ... my-app-image --param1=arg1_value

arg1_value will overwrite param1's default value (arg1_default).

If --param1 is omitted then arg1_default will be applied to param1 which will be passed to my-app executable.

sed - Passing variable from container start to file - Stack Overflow

While they seem similar, ENTRYPOINT and CMD serve different purposes and behave differently.

Key Difference:
  • ENTRYPOINT: Defines the executable that will always run. Arguments are appended to it.
  • CMD: Provides default arguments that can be completely replaced.

Practical Examples

With CMD:

dockerfile:

CMD ["s3cmd", "ls", "s3://mybucket"]

bash:

# Uses the default
docker run my-image
# Runs: s3cmd ls s3://mybucket

# Completely replaces CMD
docker run my-image echo "hello"
# Runs: echo "hello" (NOT s3cmd!)


With ENTRYPOINT:

dockerfile:

ENTRYPOINT ["s3cmd"]

bash:

# Must provide arguments
docker run my-image ls s3://mybucket
# Runs: s3cmd ls s3://mybucket

# Still runs s3cmd, just different args
docker run my-image --help
# Runs: s3cmd --help

# Even this runs s3cmd
docker run my-image echo "hello"
# Runs: s3cmd echo "hello" (probably an error!)

Best Practice: Use Both Together

dockerfile:

ENTRYPOINT ["s3cmd"]
CMD ["--help"]

This way:
  • s3cmd always runs (ENTRYPOINT)
  • If no arguments provided, it shows help (CMD as default)
  • Any arguments you provide replace CMD but still go to s3cmd

bash:

docker run my-image              # Runs: s3cmd --help
docker run my-image ls s3://bucket  # Runs: s3cmd ls s3://bucket

Use ENTRYPOINT when you want your container to behave like a specific command-line tool.
Use CMD when you want a default command that users might want to completely override.

With CMD ["s3cmd", "ls", "s3://mybucket"], you can replace all the arguments by simply providing new ones after the image name:

bash:

# Replace with completely different command
docker run my-image echo "hello"
# Runs: echo "hello"

# Replace with different s3cmd arguments
docker run my-image s3cmd sync s3://source s3://dest
# Runs: s3cmd sync s3://source s3://dest

# Replace with a shell
docker run my-image /bin/bash
# Runs: /bin/bash

Important: 

CMD, whatever we provide after the image name completely replaces the entire CMD instruction. We're not appending to it - we're replacing it entirely.

So if we want to run a different s3cmd command, we need to include s3cmd again in our docker run command because we're replacing the whole thing.

This is why for tools like s3cmd, using ENTRYPOINT is often better:

dockerfile:

ENTRYPOINT ["s3cmd"]
CMD ["ls", "s3://mybucket"]

Then we can do:

docker run my-image                    # Runs: s3cmd ls s3://mybucket
docker run my-image sync s3://a s3://b # Runs: s3cmd sync s3://a s3://b

We only need to provide the arguments, not the s3cmd command itself again.


EXPOSE


Used optionally, only for documenting and giving a hint to whoever runs "docker run" which port should be published (with -p/--publish or -P/--publish-all).

...
EXPOSE 8080
...

This means that when we run the container, we should expose this port to the outside world. We have two options for this:
  • -p, --publish - maps a host port we specify (manually assign) to a running container port
    • docker run -p local_port:container_port
    • Example: docker run -p 8080:8080
  • -P, --publish-all - publishes all exposed ports to ports that Docker randomly picks (available high-order ports, higher than 30000)
    • Example: docker run -P
In both cases a firewall rule is created which maps a container port to a port on the Docker host.

FROM

  • Set the baseImage to use for subsequent instructions
  • must be the first instruction in a Dockerfile.

FROM baseImage
FROM baseImage:tag
FROM baseImage@digest

If we don't want to use Docker Hub as Docker repository but some custom server, we can write:

FROM docker.example.com/image_name


Thursday, 10 September 2020

How to test SSH key password on Ubuntu?

 How to test password for a private SSH key?


If id_rsa and id_rsa.pub is a keypair, we can execute (after we go to directory whey they reside like e.g.  cd ~/.ssh/):

$ ssh-keygen -y -f id_rsa

...which will prompt us to enter the password. If correct, this will output the public key.

-y This option will read a private OpenSSH format file and print an OpenSSH public key to stdout.
-f filename  Specifies the filename of the key file.

If you download key pair from another machine, this operation might fail with error:

$ ssh-keygen -y -f id_rsa
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Permissions 0664 for 'id_rsa' are too open.
It is required that your private key files are NOT accessible by others.
This private key will be ignored.
Load key "id_rsa": bad permissions

To fix this change permissions on file:

$ chmod 400 id_rsa

If private key is not password protected, user will not be prompted to enter it.

 

Resources:

ssh keys - How do I verify/check/test/validate my SSH passphrase? - Stack Overflow

command line - How do I retrieve the public key from a SSH private key? - Ask Ubuntu 

How to generate SSH key pair on Ubuntu



To create SSH key pair we can use ssh-keygen:

$ ssh-keygen -t rsa -b 4096 -C "your_email@example.com"

-t Specifies the type of key to create.  The possible values are “dsa”, “ecdsa”, “ed25519”, or “rsa”.

-b bits Specifies the number of bits in the key to create. For RSA keys, the minimum size is 1024 bits and the default is 3072 bits. Generally, 3072 bits is considered sufficient

-C comment Provides a new comment. This can be any string you want to help identify the key. As this keypair is unique and represents an identity of the (e.g. repository) user, I tend to use email format: user@domain


In case of Ed25519 there is no need to set the key size, as all Ed25519 keys are 256 bits. Older SSH clients and servers may not support these keys.

$ ssh-keygen -t ed25519 -C "your_email@example.com"


Example:
 
$ ssh-keygen -t rsa -b 4096 -C "bojan@xyz.com"
Generating public/private rsa key pair.
Enter file in which to save the key (/home/bojan/.ssh/id_rsa): ./key-pair--ec2--my-app
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in ./key-pair--ec2--my-app
Your public key has been saved in ./key-pair--ec2--my-app.pub
The key fingerprint is:
SHA256:Hft9KWA0w7qalQIRUFBX3MunZ8HJqm0LPXb2P/zcKKI 
bojan@xyz.com
The key's randomart image is:
+---[RSA 4096]----+
|   o=+ .o..      |
|      o  ...     |
|     .    o=+ .  |
|      .  .o=o*   |
|     .  S.oo+ .  |
|      .   =+.+  .|
|       . =o+++o..|
|        =.+o= o*.|
|       oE..o....O|
+----[SHA256]-----+

$ ls -la
-rw------- 1 bojan bojan 3434 May 27 17:27 key-pair--ec2--my-app
-rw-r--r-- 1 bojan bojan  749 May 27 17:27 key-pair--ec2--my-app.pub



To copy the contents of the id_rsa.pub file to clipboard:

$ xclip -sel clip < ~/.ssh/id_rsa.pub


Private key with default name (~/.ssh/id_rsa) should automatically be added to the SSH authentication agent. To check this we can start the ssh-agent and 
 
$ eval "$(ssh-agent -s)"
Agent pid 76155

$ ssh-add -l -E sha256
256 SHA256:DUXxZAyhbh68kJwex8rzHXQM2cKzSWadNqzW1KnPR3A bojan@xyz.com (ED25519)

If we have only key that does not have the default name, it might not have been added to SSH agent in which case the output would be:

$ ssh-add -l -E sha256
The agent has no identities.

To add the key to SSH agent:

$ ssh-add path/to/mykey
Enter passphrase for path/to/mykey: 
Identity added: path/to/mykey (bojan@xyz.com)


To push public key to the remote Linux machine:



References


Friday, 21 August 2020

How to upgrade Go on Ubuntu

I had oldish version (1.12.1) of Go on my Ubuntu 18.04 and I wanted to update it to the most recent one (1.15). 



This was my old version:

$ go version
go version go1.12.1 linux/amd64

It was using symlink pointing to my previous go installation which was in /var/lib/go:

$ which go /usr/local/bin/go $ ls -la /usr/local/bin/go lrwxrwxrwx 1 root root 18 Apr 1 2019 /usr/local/bin/go -> /var/lib/go/bin/go $ ls -la /var/lib/go/bin/go -rwxr-xr-x 1 root root 14609408 Mar 14 2019 /var/lib/go/bin/go $ ls -la /var/lib/go total 216 drwxr-xr-x 10 root root 4096 Mar 14 2019 . drwxr-xr-x 78 root root 4096 May 26 10:02 .. drwxr-xr-x 2 root root 4096 Mar 14 2019 api -rw-r--r-- 1 root root 55358 Mar 14 2019 AUTHORS drwxr-xr-x 2 root root 4096 Mar 14 2019 bin -rw-r--r-- 1 root root 1339 Mar 14 2019 CONTRIBUTING.md -rw-r--r-- 1 root root 78132 Mar 14 2019 CONTRIBUTORS drwxr-xr-x 8 root root 4096 Mar 14 2019 doc -rw-r--r-- 1 root root 5686 Mar 14 2019 favicon.ico drwxr-xr-x 3 root root 4096 Mar 14 2019 lib -rw-r--r-- 1 root root 1479 Mar 14 2019 LICENSE drwxr-xr-x 13 root root 4096 Mar 14 2019 misc -rw-r--r-- 1 root root 1303 Mar 14 2019 PATENTS drwxr-xr-x 6 root root 4096 Mar 14 2019 pkg -rw-r--r-- 1 root root 1607 Mar 14 2019 README.md -rw-r--r-- 1 root root 26 Mar 14 2019 robots.txt drwxr-xr-x 46 root root 4096 Mar 14 2019 src drwxr-xr-x 21 root root 12288 Mar 14 2019 test -rw-r--r-- 1 root root 8 Mar 14 2019 VERSION

I then removed the old installation:

$ sudo rm -rf /var/lib/go/

...and downloaded and unpacked the archive containing the latest version at recommended locaton (/usr/local):

$ sudo tar -C /usr/local -xzf go1.15.linux-amd64.tar.gz

$ ls -la /usr/local/go
total 236
drwxr-xr-x 10 root root  4096 Aug 11 20:16 .
drwxr-xr-x 12 root root  4096 Aug 21 13:14 ..
drwxr-xr-x  2 root root  4096 Aug 11 20:16 api
-rw-r--r--  1 root root 55669 Aug 11 20:16 AUTHORS
drwxr-xr-x  2 root root  4096 Aug 11 20:18 bin
-rw-r--r--  1 root root  1339 Aug 11 20:16 CONTRIBUTING.md
-rw-r--r--  1 root root 95475 Aug 11 20:16 CONTRIBUTORS
drwxr-xr-x  7 root root  4096 Aug 11 20:16 doc
-rw-r--r--  1 root root  5686 Aug 11 20:16 favicon.ico
drwxr-xr-x  3 root root  4096 Aug 11 20:16 lib
-rw-r--r--  1 root root  1479 Aug 11 20:16 LICENSE
drwxr-xr-x 12 root root  4096 Aug 11 20:16 misc
-rw-r--r--  1 root root  1303 Aug 11 20:16 PATENTS
drwxr-xr-x  6 root root  4096 Aug 11 20:19 pkg
-rw-r--r--  1 root root  1607 Aug 11 20:16 README.md
-rw-r--r--  1 root root    26 Aug 11 20:16 robots.txt
-rw-r--r--  1 root root   397 Aug 11 20:16 SECURITY.md
drwxr-xr-x 47 root root  4096 Aug 11 20:16 src
drwxr-xr-x 23 root root 12288 Aug 11 20:16 test
-rw-r--r--  1 root root     6 Aug 11 20:16 VERSION

$ cat /usr/local/go/VERSION 
go1.15

As expected, go binary was not available at the moment for the broken symlink:

$ go version
bash: /usr/local/bin/go: No such file or directory

$ ls -la /usr/local/bin/go
lrwxrwxrwx 1 root root 18 Apr  1  2019 /usr/local/bin/go -> /var/lib/go/bin/go // (symlink error)

I then deleted the old symlink and created a new one:

$ sudo rm /usr/local/bin/go 
$ sudo ln -s /usr/local/go/bin/go /usr/local/bin/go

After this my go symlink was pointing to the new version:

$ go version
go version go1.15 linux/amd64

$ which go
/usr/local/bin/go

$ ls -la /usr/local/bin/go
lrwxrwxrwx 1 root root 20 Aug 21 13:20 /usr/local/bin/go -> /usr/local/go/bin/go

$ ls -la /usr/local/go/bin/go
-rwxr-xr-x 1 root root 14256244 Aug 11 20:18 /usr/local/go/bin/go

$ /usr/local/go/bin/go version
go version go1.15 linux/amd64

Friday, 17 July 2020

Functions in Go


To declare a function in Go, use the keyword func followed by the function name, any
parameters, and then any return values.

feeds, err := RetrieveFeeds()
if err != nil {
   log.Fatal(err)
}

NOTE: You can omit the parentheses () from an if statement in Golang, but the curly braces {} are mandatory!

This function belongs to the search package and returns two values. The first return value is a slice
of Feed type values. A slice is a reference type that implements a dynamic array. You use
slices in Go to work with lists of data.

The second return value is an error.  
Functions can have multiple return values. It’s common to declare functions that return a value and an error value just like the RetrieveFeeds function. If an error occurs, never trust the other values being returned from the function. They should always be ignored, or else you run the risk of the code generating more errors or panics.

Why doesn't Go allow nested function declarations (functions inside functions)? - Stack Overflow


func plus(a int, b int) int {...}

Can Functions be passed as parameters in Go? Yes.

type MyFunc func(int) string

func Foo(fn MyFunc) {
    str := fn(123)
}

Variable can be of a function type and be assigned a function:

sayHello := func() {
   fmt.Println("hello")
}

go sayHello()


Go by Example: Variadic Functions

func sum(nums... int){...}

Three dots (ellipsis) notation

Specifying function's default value for an argument is NOT supported.
Default value in Go's method

When to return pointer to local variable?

Return pointer to local struct

Go performs pointer escape analysis. If the pointer escapes the local stack, the object is allocated on the heap. If it doesn't escape the local function, the compiler is free to allocate it on the stack (although it makes no guarantees; it depends on whether the pointer escape analysis can prove that the pointer stays local to this function).

Golang documentation states that it's perfectly legal to return a pointer to local variable.
Compiler sees you return the address and just makes it on the heap for you.

Escape Analysis in Go


Naming


Your custom function can't have the same name as the name of some imported package.

Function Overloading


Does the Go language have function/method overloading? (Answer: NO)
Optional parameters, default parameter values and method overloading

The idiomatic way to emulate optional parameters and method overloading in Go is to write several methods with different names.

Alternative Patterns for Method Overloading in Go
Functional options for friendly APIs <-- MUST READ (!)

Deferred execution

go - Multiple defers vs deferred anonymous function - Stack Overflow

defer func() {
  err := f.Close()
  if err != nil {
    log.Println("close:", err)
  }
  err = os.Remove(f.Name())
  if err != nil {
    log.Println("remove:", err)
  }
}()

Built-in Functions


make()


panic()


Go by Example: Panic
When to call panic() : golang
Panic in Golang - GeeksforGeeks


Closures


Go by Example: Closures

What exactly does “closing over” mean?

 "A closes over B" == "B is a free variable of A", where free variables are those that appear in a function's body, but not in its signature.


What will be the output of the following snippet?

var wg sync.WaitGroup

for _, letter := range []string{"a", "b", "c"} {
   wg.Add(1)
   go func() {
      defer wg.Done()
      fmt.Println(letter)
   }()
}

wg.Wait()


Go closure variable scope

Closures in Go capture variables by reference. That means the inner function holds a reference to the  i variable in the outer scope, and each call of it accesses this same variable.


Closure (computer programming)

Comapre to JS Clsoures.
Lexical scoping:
This is an example of lexical scoping, which describes how a parser resolves variable names when functions are nested. The word "lexical" refers to the fact that lexical scoping uses the location where a variable is declared within the source code to determine where that variable is available. Nested functions have access to variables declared in their outer scope.


How golang's “defer” capture closure's parameter?



// Launch the goroutine
go func(matcher Matcher, feed *Feed) {
   Match(matcher, feed, searchTerm, results) 
   waitGroup.Done()
}(matcher, feed)


goroutine

  • light-weight process that is automatically time-sliced onto one or more operating system threads by the Go runtime.
  • a function that’s launched to run independently from other functions in the program.
  • Use the keyword go to launch and schedule goroutines to run concurrently. 
  • the order the goroutines get executed is unpredictable
  • can be an anonymous function
  • can be launched in a (for-range) loop, for each element of some set: this allows each element to be processed independently in a concurrent fashion
  • There's no goroutine ID available from the runtime
  • not an OS thread (thread managed/scheduled natively by OS)
  • not exactly a green thread (thread managed/scheduled by languages runtime or virtual machine) [Green threads] [Why not Green Threads?]
  • is a special type of coroutine (concurrent subroutines - functions, closures or methods) - the one that is non-preemptive. It cannot be interrupted but instead has multiple points through which it can be suspended or reentered. Go runtime defines this points internally and automatically suspends them when they block and resumes them when they become unblocked.
  • goroutines operate within the same address space as each other, and host functions
  • Go implements M:N scheduler, which means it maps M green threads to N OS threads. Goroutines are then scheduled onto the green threads. When we have more goroutines than green threads available, the scheduler handles the distribution of the goroutines across the available threads and ensures that when these goroutines become blocked, other goroutines can be run.
  • Go follows a model of concurrency called the fork-join model.
  • function which runs as goroutine can have return value(s) and return them but they will never be read => they should not have a return value. If we want to read some value which is calculated in goroutine we should either use channels or variable capturing mechanisms [Catching return values from goroutines].

To launch a function as goroutine:
goroutines operate within the same address space as each other, and
simply host functions,
func foo() {
   fmt.Println("foo()")
}

go foo()

To launch anonymous function as goroutine:

go func() {
   fmt.Println("foo()")
}()



anonymous function

  • a function that’s declared without a name
  • can take parameters


pointer variables

  • are great for sharing variables between functions. They allow functions to access and change the state of a variable that was declared within the scope of a different function and possibly a different goroutine.
  • In Go, all variables are passed by value. Since the value of a pointer variable is the address to the memory being pointed to, passing pointer variables between functions is still considered a pass by value.
Pointers vs. values in parameters and return values


waitGroup.Done()

Once the main task withing the goroutine completes, we execute the code which decrements the WaitGroup count. Once every goroutine finishes calling Done method, the program will know every main task has been done (e.g. element has been processed).

There’s something else interesting about the method call to Done: the WaitGroup
value was never passed into the anonymous function as a parameter, yet the anony-
mous function has access to it. Go supports closures and you’re seeing this in action. In fact, the searchTerm and results variables are also being accessed by the anonymous function via closures.
Thanks to closures, the function can access those variables directly without the need to
pass them in as parameters. The anonymous function isn’t given a copy of these variables; it has direct access to the same variables declared in the scope of the outer function. This is the reason why we don’t use closures for the matcher and feed variables.

---

With all the processing goroutines working, sending results on the results channel
and decrementing the waitGroup counter, we need a way to display those results and
keep the main function alive until all the processing is done. We'll launch yet another anonymous function as a goroutine. This anonymous function takes no parameters and uses closures to access both the waitGroup and results variables. This goroutine calls the method Wait() on the WaitGroup value, which is causing the goroutine to block until the count for the WaitGroup hits zero. Once that happens, the goroutine calls the built-in function close on the channel, which as you’ll see causes the program to terminate.


// Launch a goroutine to monitor when all the work is done.
go func() {
   // Wait for everything to be processed.
   waitGroup.Wait()

   // Close the channel to signal to the Display

   // function that we can exit the program.
   close(results)
}()

---

func Match(matcher Matcher, feed *Feed, searchTerm string, results chan<- *Result) {
   // Perform the search against the specified matcher.
   searchResults, err := matcher.Search(feed, searchTerm)
   if err != nil {
      log.Println(err)
      return
   }
   // Write the results to the channel.
   for _, result := range searchResults {
      results <- result
   }
}

results are written to the channel

BK: not how nil is used instead of null
BK: note how it's so convenient the model of returning function result and error at the same time; no need to pass variable by pointer and no expensive exceptions
BK: note how there are no brackets around conditions in if, for... statements.

init() function


All init functions in any code file that are part of the program will get called before
the main function.

When is the init() function run?

func init() {
  // Change the device for logging to stdout.
   log.SetOutput(os.Stdout)
   ...
}

This init function sets the logger from the standard library to write to the stdout device. By default, the logger is set to write to the stderr device.


main() function


How do I return from func main in Go? - Stack Overflow

func main() { os.Exit(mainReturnWithCode()) }

func mainReturnWithCode() int {
    // do stuff, defer functions, etc.
    return exitcode // a suitable exit code
}

Working with strings in Go

Strings are implemented as a reference type, though they're immutable.

String 


s := "This is in first line"
s += "\n"
s += "...and this is in the second line"

How do you write multiline strings in Go?

This raw quote (raw literal) does not parse escape sequences (\n would remain):

`line 1
line 2\n
line 3`

It is possible to use formatters though:

fmt.Sprintf(`a = %d`, 123)

Another option:

"line 1" +
"line 2" +
"line 3"



What is the difference between backticks (``) & double quotes (“”) in golang?

String Comparison - use == or !=

Slice string into letters

String Formatting 


s := fmt.Sprintf("a %s", "string")

Go by Example: String Formatting
Extracting substrings in Go

%x - formats value as HEX string with lowercase letters
%X - formats value as HEX string with uppercase letters

var n []byte = ...
s := fmt.Sprinf("%x", n) 

How to print struct variables in console? To print the name of the fields in a struct:

fmt.Printf("%+v\n", myStruct)


To convert string to byte array use:

s string
buffer := []byte(s)


fmt


fmt.Println("Table names:", tableNames)

SPACE character is automatically inserted between these two strings.

Golang - How to print the values of Arrays?

fmt.Printf("%v", projects)


fmt.Printf


%+v - prints struct’s field names (if value is a struct)





Go string = byte sequence, UTF-8 encoded
Unicode UTF-8 code point = 1-4 bytes (1 for ASCII)
rune = int32 => can store each UTF-8 code point 

Unicode string example: en dash & Chinese letter 

package main

import (
"fmt"
)

func main() {
s := "\u2013汉"
fmt.Printf("Character %s has %d bytes, %d UTF-8 code points", s, len(s), len([]rune(s)))
}

Output:

Character –汉 has 6 bytes, 2 UTF-8 code points