Showing posts with label Windows. Show all posts
Showing posts with label Windows. Show all posts

Saturday, 22 December 2018

Setting up Go on Windows

Go (Golang) is installed on Windows via its msi installer available on Go's website. The latest version at the time of writing this article was 1.11.4 and the installer go1.11.4.windows-amd64.msi.



Installation is straightforward:







By default, installer creates or modifies the following environment variables:

In System space:

   Variable (created): GOROOT
   Value: C:\Go\
   Description: Go installation location.

   Variable (value appended): Path
   Value: C:\Go\bin
   Description: Go binaries location.


In User space:

   Variable (created): GOPATH
   Value: %USERPROFILE%\go
   Description: User's go workspace path.

   Variable (value appended): Path
   Value: %USERPROFILE%\go\bin
   Description: User's Go applications' binaries location.


Go workspace:
  • a directory with two subdirectories: bin and src
    • src typically contains all repositories with Go projects
    • bin contains built and then installed Go application binaries
  • can be set at arbitrary location; I set mine to C:\dev\go (and also set User space Path to C:\dev\go\bin)

Upon installation, C:\Go\bin contains three binaries:
  • go -  Tool for managing Go source code
  • godoc - Documentation tool; parses Go source code - including comments - and produces documentation as HTML or plain text
  • gofmt - Go source code formatter; uses tabs for indentation and blanks for alignment. 



Let's see what are the command line arguments of go tool:

>go
Go is a tool for managing Go source code.

Usage:

        go <command> [arguments]

The commands are:

        bug         start a bug report
        build       compile packages and dependencies
        clean       remove object files and cached files
        doc         show documentation for package or symbol
        env         print Go environment information
        fix         update packages to use new APIs
        fmt         gofmt (reformat) package sources
        generate    generate Go files by processing source
        get         download and install packages and dependencies
        install     compile and install packages and dependencies
        list        list packages or modules
        mod         module maintenance
        run         compile and run Go program
        test        test packages
        tool        run specified go tool
        version     print Go version
        vet         report likely mistakes in packages

Use "go help <command>" for more information about a command.

Additional help topics:

        buildmode   build modes
        c           calling between Go and C
        cache       build and test caching
        environment environment variables
        filetype    file types
        go.mod      the go.mod file
        gopath      GOPATH environment variable
        gopath-get  legacy GOPATH go get
        goproxy     module proxy protocol
        importpath  import path syntax
        modules     modules, module versions, and more
        module-get  module-aware go get
        packages    package lists and patterns
        testflag    testing flags
        testfunc    testing functions





If we already have some Go project in some Git repository (like GitHub), we can use get command to clone the given repository:

C:\dev\go\src>go get github.com/BojanKomazec/go-hello-world
package github.com/BojanKomazec/go-hello-world: no Go files in
C:\dev\go\src\github.com\BojanKomazec\go-hello-world

Repository is cloned to C:\dev\go\src\github.com\BojanKomazec\go-hello-world directory.

Repository shall be written without starting https:// as go get would report an error:

C:\dev\go\src>go get https://github.com/BojanKomazec/go-hello-world
package https:/github.com/BojanKomazec/go-hello-world: https:/github.com/BojanKomazec/go-hello-world: invalid import path: malformed import path "https:/github.com/BojanKomazec/go-hello-world": invalid char ':'

This my repository only has .gitignore and README.md files so let's add some code. First, let's create a directory named hello and in it a go file with the following content:

C:\dev\go\src\github.com\BojanKomazec\go-hello-world\hello\hello.go:

package main

import "fmt"

func main() {
   fmt.Printf("hello, world\n")
}


Building this file with:

C:\dev\go\src\github.com\BojanKomazec\go-hello-world\hello>go build

...creates a hello.exe file in the same directory and we can run it:

C:\dev\go\src\github.com\BojanKomazec\go-hello-world\hello>hello
hello, world

To install that build as a package, we have to run:

C:\dev\go\src\github.com\BojanKomazec\go-hello-world\hello>go install

This deploys application binary in C:\dev\go\bin directory.

go tool and Git

If we add, commit and then try to push hello.go file to remote repository, we'll be prompted by Git to enter our GitHub credentials as by default go get sets the https-based URL of the remote. We can verify that with:

C:\dev\go\src\github.com\BojanKomazec\go-hello-world>git remote show origin
* remote origin
  Fetch URL: https://github.com/BojanKomazec/go-hello-world
  Push  URL: https://github.com/BojanKomazec/go-hello-world
  HEAD branch: master
  Remote branch:
    master tracked
  Local branch configured for 'git pull':
    master merges with remote master
  Local ref configured for 'git push':
    master pushes to master (up to date)

We can set SSH-based URL by executing:

>git remote set-url origin git@github.com:BojanKomazec/go-hello-world.git

Let's verify it:

C:\dev\go\src\github.com\BojanKomazec\go-hello-world>git remote show origin
Enter passphrase for key '/c/Users/komazec/.ssh/id_rsa':
* remote origin
  Fetch URL: git@github.com:BojanKomazec/go-hello-world.git
  Push  URL: git@github.com:BojanKomazec/go-hello-world.git
  HEAD branch: master
  Remote branch:
    master tracked
  Local branch configured for 'git pull':
    master merges with remote master
  Local ref configured for 'git push':
    master pushes to master (up to date)

Let's now analyse code in the go source file above.

package main

func main() {
   ...
}


This line tells the Go compiler that the package should compile as an executable program. Its entry point will be function main(). If we were building a shared library, we would use package lib directive and there would not be main() function in the package.

import "fmt"
...
fmt.Printf("hello, world\n")

import statement imports a package into other package. Here, we wanted to use function which prints a string onto standard output. Such function, Printf, is available in the fmt package which comes from the Go standard library. Go compiler looks for standard library packages on paths specified in GOROOT and your and third-party packages on path specified in GOPATH environment variables.

Project on GutHub:

https://github.com/BojanKomazec/go-hello-world

References:

golang.org: Go installation on Windows
Understanding Golang Packages
Package “main” and func “main”

Thursday, 18 October 2018

How to install Node.js on Windows

Download the latest Node installer (node-v6.11.4-x64.msi at the time of writing) and execute it.
Upon installation system variable Path (in environment variables) contains a new entry:
C:\Program Files\nodejs\.

Let's see what's inside this directory:

c:\Program Files\nodejs\node_modules\
c:\Program Files\nodejs\node_modules\npm\
c:\Program Files\nodejs\node_modules\npm\.github\
c:\Program Files\nodejs\node_modules\npm\bin\
c:\Program Files\nodejs\node_modules\npm\bin\node-gyp-bin\
c:\Program Files\nodejs\node_modules\npm\bin\npm
c:\Program Files\nodejs\node_modules\npm\bin\npm.cmd
c:\Program Files\nodejs\node_modules\npm\bin\npm-cli.js
c:\Program Files\nodejs\node_modules\npm\bin\read-package-json.js

c:\Program Files\nodejs\node_modules\npm\changelogs\
c:\Program Files\nodejs\node_modules\npm\doc\
c:\Program Files\nodejs\node_modules\npm\html\
c:\Program Files\nodejs\node_modules\npm\lib\
c:\Program Files\nodejs\node_modules\npm\man\
c:\Program Files\nodejs\node_modules\npm\node_modules\
c:\Program Files\nodejs\node_modules\npm\scripts\
c:\Program Files\nodejs\node_modules\npm\test\
c:\Program Files\nodejs\node_modules\npm\.mailmap
c:\Program Files\nodejs\node_modules\npm\.npmignore
c:\Program Files\nodejs\node_modules\npm\AUTHORS
c:\Program Files\nodejs\node_modules\npm\configure
c:\Program Files\nodejs\node_modules\npm\LICENSE
c:\Program Files\nodejs\node_modules\npm\Makefile
c:\Program Files\nodejs\node_modules\npm\npmrc
c:\Program Files\nodejs\node_modules\npm\make.bat
c:\Program Files\nodejs\node_modules\npm\cli.js
c:\Program Files\nodejs\node_modules\npm\package.json
c:\Program Files\nodejs\node_modules\npm\CHANGELOG.md
c:\Program Files\nodejs\node_modules\npm\CONTRIBUTING.md
c:\Program Files\nodejs\node_modules\npm\README.md
c:\Program Files\nodejs\node_modules\npm\.travis.yml
c:\Program Files\nodejs\node_modules\npm\appveyor.yml

c:\Program Files\nodejs\npm
c:\Program Files\nodejs\nodevars.bat
c:\Program Files\nodejs\npm.cmd
c:\Program Files\nodejs\node.exe
c:\Program Files\nodejs\node_etw_provider.man
c:\Program Files\nodejs\node_perfctr_provider.man

How to check installed version?

>node --version
v10.7.0

Thursday, 13 September 2018

node-gyp and Python support on Windows

node-gyp is a cross-platform command-line tool written in Node.js for compiling native addon modules for Node.js. [node-gyp on GitHub] [node-gyp]

If you try to run node-gyp on a machine with no Python installed you might get an error similar to this:

gyp ERR! configure error
gyp ERR! stack Error: Can't find Python executable "python", you can set the PYTHON env variable.
gyp ERR! stack at PythonFinder.failNoPython (C:\Users\user\AppData\Roaming\npm\node_modules\npm\node_modules\node-gyp\lib\configure.js:484:19)
gyp ERR! stack at PythonFinder. (C:\Users\user\AppData\Roaming\npm\node_modules\npm\node_modules\node-gyp\lib\configure.js:509:16)
gyp ERR! stack at C:\Users\user\AppData\Roaming\npm\node_modules\npm\node_modules\graceful-fs\polyfills.js:284:29
gyp ERR! stack at FSReqWrap.oncomplete (fs.js:158:21)
gyp ERR! System Windows_NT 10.0.17134
gyp ERR! command "C:\\Program Files\\nodejs\\node.exe" "C:\\Users\\user\\AppData\\Roaming\\npm\\node_modules\\npm\\node_modules\\node-gyp\\bin\\node-gyp.js" "rebuild"
gyp ERR! cwd C:\whatever\my-app\node_modules\dtrace-provider
gyp ERR! node -v v10.7.0
gyp ERR! node-gyp -v v3.8.0
gyp ERR! not ok

(cwd in this output message stands for Current Working Directory)

If you have Python 3 installed, the same command will fail for Python 3 interpreter not be able to parse Python 2 code...This happens as node-gyp (still) doesn't support Python 3 so you'll need to install Python 2 (2.7.x).

When installing Python 2 DO NOT opt for adding its path to PATH environment variable as otherwise it will make Python 2 the default one on the machine.

The easiest way to make node-gyp working with Python 2 while not interfering with existing Python 3 paths is to follow the suggestion from the error message above, create PYTHON env variable and set its value to Python 2 path (C:\Python27). Then we can have this setup before we execute node node-gyp.js:

>python --version
Python 3.7.0

>echo %PYTHON%
C:\Python27


How to install Python on Windows?

How to install the latest version of Python?

  • Go to Python Releases for Windows page and click the link for the latest release, e.g. Latest Python 3 Release - Python 3.7.0
  • Scroll to the bottom of the page and click on the link of the installer to download it. I opted for "Windows x86-64 web-based installer".
  • Run the installer. On the first page of the installer opt for adding Python to PATH.
  • Python will be installed in C:\Users\User\AppData\Local\Programs\Python\Python37.

If you haven't opted for adding Python to PATH during installation, you open a Terminal and try to run Python interpreter by typing python you'll get:

>python
'python' is not recognized as an internal or external command,
operable program or batch file.

To resolve this and make python binary visible from every directory add its path to Path environment variable. After that, python interpreter is visible from anywhere (in a new Terminal session):

>python
Python 3.7.0 (v3.7.0:1bf9cc5093, Jun 27 2018, 04:59:51) [MSC v.1914 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>>


How to install multiple versions of Python on the same system?


Sometimes you need to have on your machine installed various versions of Python e.g. some applications might require Python 2 and some Python 3. I already installed Python 3 above and now I'm gonna install Python 2:
  • Go to Python Releases for Windows page and click the link for the latest release, e.g. Latest Python 2 Release - Python 2.7.15
  • Scroll to the bottom of the page and click on the link of the installer to download it. I opted for "Windows x86-64 MSI installer".
  • Run the installer. On the first page of the installer you can opt for adding Python 2 to PATH.
  • Python will be installed in C:\Python27.
If you opt for adding Python 2 to Path, restart the Terminal (so it can pick up the latest values of environment variables) and type python, Terminal will pick Python 2:

>python
Python 2.7.15 (v2.7.15:ca079a3ea3, Apr 30 2018, 16:30:26) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>>

Python 2 installer has added C:\Python27 (and C:\Python27\Scripts) as first entries in PATH variable so Terminal now picks python.exe from this directory.

To make sure you are using desired version of Python interpreter in Terminal the best is to remove all Python paths from PATH and then set path to desired Python version for the current Terminal session (temporarily) by executing:

>set PATH=path\to\desired\python;%PATH%

Thursday, 26 October 2017

How to save paths from Search results in Windows Explorer

From time to time I want to document location(s) of some file on my system. I usually use Total Commander which has built-in functionality of copying the paths of selected files but it is possible to achieve the same in good old Windows Explorer. Once search is complete and all results are displayed, select them all with CTRL+A, hold down SHIFT key and do the right click anywhere on the selection. In the context menu which appears, find and click on Copy as path. All paths are now copied in the clipboard.


Now I can paste all paths:


"C:\Windows\SysWOW64\ucrtbase.dll"
"C:\Windows\WinSxS\wow64_microsoft-windows-ucrt_31bf3856ad364e35_10.0.16299.15_none_d9020b8bbf051ead\ucrtbase.dll"
"C:\Windows\System32\ucrtbase.dll"
"C:\Windows\WinSxS\amd64_microsoft-windows-ucrt_31bf3856ad364e35_10.0.16299.15_none_cead61398aa45cb2\ucrtbase.dll"
"C:\Windows\System32\ucrtbase_enclave.dll"
"C:\Windows\WinSxS\amd64_microsoft-onecore-i..atedusermode-common_31bf3856ad364e35_10.0.16299.15_none_215b207fb180b0b9\ucrtbase_enclave.dll"
"C:\Program Files (x86)\Microsoft Visual Studio 15.0\Team Tools\Performance Tools\ucrtbase.dll"
"C:\Program Files\Microsoft Visual Studio 15.0\Common7\IDE\Remote Debugger\x86\ucrtbase.dll"
"C:\Program Files (x86)\Microsoft Visual Studio 15.0\Team Tools\Performance Tools\x64\ucrtbase.dll"
"C:\Program Files\Microsoft Visual Studio 15.0\Common7\IDE\Remote Debugger\x64\ucrtbase.dll"
"C:\Program Files\Microsoft Visual Studio 15.0\Remote Tools\DiagnosticsHub\ucrtbase.dll"
"C:\Program Files\Microsoft Visual Studio 15.0\Team Tools\DiagnosticsHub\Collector\ucrtbase.dll"
"C:\Windows\WinSxS\Backup\wow64_microsoft-windows-ucrt_31bf3856ad364e35_10.0.16299.15_none_d9020b8bbf051ead_ucrtbase.dll_a00b9625"
"C:\Windows\WinSxS\Backup\amd64_microsoft-windows-ucrt_31bf3856ad364e35_10.0.16299.15_none_cead61398aa45cb2_ucrtbase.dll_a00b9625"
"C:\Users\bojan\AppData\Local\Microsoft\OneDrive\17.3.7073.1013\ucrtbase.dll"

Tuesday, 12 January 2016

Sniffing the traffic on the loopback interface on Windows with RawCap

If you ever had to write a chunk of code for inter-process communication via Internet Protocol (IP) you must have came across using the loopback interfaceIPv4 address 127.0.0.1 is usually the address of this virtual interface and also the address that hostname localhost  resolves to. You would, at least for tests, set one process (server) listening on some port on the localhost while the other process (client) would connect to the server and start their data exchange. 

In order to check the data sent from one end to another, we can use any kind of process output (standard output, log, ...) provided that given process actually shows that data. But what if there is no such output? We have to find a way how to sniff the traffic between these two parties.

RawCap is one of free tools capable of sniffing packets on the loopback interface. It collects packets and stores them in the file in pcap format. This file can be opened in Wireshark.

In order to demonstrate RawCap packet capturing we can use ready-made networking application which can serve both as a server and a client - Ncat. It comes as a part of Nmap for Windows and uses TCP by default.

Once Nmap is installed, open console window in C:\Program Files (x86)\Nmap directory and  type the following:


> ncat -v -4 -l localhost 6789


-v (or --verbose) sets verbose output
-4 instructs ncat to use IPv4 addresses only
-l (or --listen) switch makes Ncat to listen for incoming connections on the provided hostname and port. localhost will be resolved to 127.0.0.1 and I also used an arbitrary port which was free - 6789.

We effectively have server running now but before we run the client it is necessary to start capturing packets. From the directory with RawCap.exe we have to open terminal window with Administrator's privileges and run RawCap.exe (which has to be run as Administrator). Now we just have to follow the instructions. RawCap lists network interfaces and asks which interface shall be monitored. We have to type in the number next to the loopback interface. In the next step we can set the name of the packet dump file or just leave default name which is dumpfile.pcap. At this point we have started sniffing packets on the localhost!


> RawCap.exe
Interfaces:
0. 169.xxx.xxx.xxx Local Area Connection Ethernet
1. 169.xxx.xxx.xxx Local Area Connection* 2 Wireless80211
2. 169.xxx.xxx.xxx Ethernet Ethernet
3. 169.xxx.xxx.xxx Ethernet 2 Ethernet
4. 169.xxx.xxx.xxx Local Area Connection 2 Ethernet
5. 127.0.0.1 Loopback Pseudo-Interface 1 Loopback
6. 192.xxx.xxx.xxx Wireless Network Connection 2 Wireless80211
Select interface to sniff [default '0']: 5
Output path or filename [default 'dumpfile.pcap']:
Sniffing IP : 127.0.0.1
File : dumpfile.pcap
Packets : 0


Let's now start the client. We have to open another terminal window in Nmap's directory and run another instance of Ncat which will try to establish TCP connection with the localhost:6789 endpoint:


> ncat -v 127.0.0.1 6789


Both server and client show the state of their current connections:

Server:

> ncat -v -4 -l localhost 6789
Ncat: Version 7.01 ( https://nmap.org/ncat )
Ncat: Listening on 127.0.0.1:6789
Ncat: Connection from 127.0.0.1.
Ncat: Connection from 127.0.0.1:9323.



Client:

> ncat -v 127.0.0.1 6789
Ncat: Version 7.01 ( https://nmap.org/ncat )
Ncat: Connected to 127.0.0.1:6789.



So far, only TCP handshake has taken place in the communication between these two instances of Ncat. We can expect that RawCap has captured at least these 3 packets so far (SYN, SYN-ACK, ACK). Let's now send some user data. By default Ncat works as an echo server which means that it echoes any text message it receives. If we type in the client "hello from client", that string will appear on the server's output. The same happens if we go the other way round and send message from the server to the client. Once we want to stop communication, we can use CTRL-C combination to stop one process. The other will get disconnected and also terminate. We can use the same combination in order to stop RawCap.exe.

At the end of the session we have:

Server:


Client:


RawCap.exe:



RawCap.exe saved all captured packets in file dumpfile.pcap. If we open this file in Wireshark and filter out all packets with TCP port 6789 we can see packets exchanged between our Ncat processes:


If we follow that TCP stream we can see exact text messages sent between client and server:


Monday, 30 January 2012

Thread and process synchronisation with semaphores

Semaphore is a synchronisation object that controls resource access (critical section execution) by maintaining the number (count, n) of accessors (threads or processes) (still) allowed to access the resource. While mutex strictly limits access to a single accessor at a time, semaphore allows up to N (N > 0) parallel accessors. N is defined when semaphore is being created and it represents maximum possible value of the semaphore count.

Semaphore count is changed through following operations:
  • wait() - which, on return,  decreases count on successful return (minimum is 0)
  • release() - which increases count (maximum is N)

Semaphore has two states:
  • signalled - when 0 < n <= N; wait() does not block
  • non-signalled - when n == 0; wait() blocks till semaphore gets signalled (or returns on expired timeout)

There are two types of semaphores:
  • counting - based on the count 0 <= n <= N where N > 1
  • binary - (specialisation of counting) where N == 1; when in signalled state we say it is unlocked; when in non-signalled state we say it is locked

Semaphores are signalling accessors the right of way - just as traffic semaphores, but unlike them, accessors themselves are controlling when the light for others will become green - through wait() and release() operations. Accessors are wait()-ing for a semaphore. If semaphore is signalled (n > 0), wait() returns immediately, decreasing semaphore count. When accessor (this one or any other) is finished with the shared resource it calls release() on the semaphore, increasing its count. Accessor will block on wait()-ing if semaphore is non-signalled (when maximum allowed number of accessors are sharing the resource). As soon as some accessor finishes the work and releases the semaphore, accessor's wait() will unblock and it will be allowed to access the resource.

Obviously, if N is set to 1, semaphore (called binary semaphore in this case) logically behaves like a mutex, allowing only one accessor at a time. There is a difference between two of them though: only accessor that locked the mutex can unlock it (mutex is owned by the accessor), but any accessor can release semaphore.

Majority of semaphore examples on the internet are focused on consumer-producer problem. I wanted to show use of semaphore on the example of traffic control - something that resembles the real semaphore. So, let's say we have three single-lane, one way roads joining just before a bridge which is one way but has two lanes. To reduce congestion on the bridge, traffic from only two access roads is allowed at a time. There is a semaphore with the red and green light by the each road and once it turns green, it remains in that state for some time period T. First two opened roads get green light first. As soon as the timeout expires for one of those roads and semaphore shows red, semaphore will show green for traffic that has been waiting at the third road.

Semaphore-example

We can think of roads (traffic on them) as accessors and the bridge as a resource: two roads can lead traffic to the bridge at the same time (two accessors are allowed to access shared resource). Obviously, we will set the maximum value for the semaphore count to 2 in our model. Thread will wait() as long as count is 0 but as soon as it gets increased to 1, wait() returns (decreasing count to 0 again).

This example aims to show how semaphore limits parallel access to the shared resource and how accessors (threads in this case) themselves control semaphore by wait()-ing for the semaphore and release()-ing it.

To stop threads I tend to use event object - not a flag (volatile bool variable). They are thread-safe and thread callbacks don't need to return with delay of one additional wait() cycle in the case when termination has been requested.

I wrapped event and semaphore objects (handles) into RAII-compliant classes - CScopedEvent and CScopedSemaphore.

main.cpp:




Output:

Road 1 opened
0 road(s) is(are) generating traffic...
Road 1 got green light for the next 10 seconds. Generating traffic...
1 road(s) is(are) generating traffic...
Road 2 opened
1 road(s) is(are) generating traffic...
Road 2 got green light for the next 10 seconds. Generating traffic...
2 road(s) is(are) generating traffic...
Road 3 opened
2 road(s) is(are) generating traffic...

Road 1 got red light. Waiting for a green light...
1 road(s) is(are) generating traffic...
Road 3 got green light for the next 10 seconds. Generating traffic...
2 road(s) is(are) generating traffic...

Road 2 got red light. Waiting for a green light...
1 road(s) is(are) generating traffic...
Road 1 got green light for the next 10 seconds. Generating traffic...
2 road(s) is(are) generating traffic...

Road 3 got red light. Waiting for a green light...
1 road(s) is(are) generating traffic...
Road 2 got green light for the next 10 seconds. Generating traffic...
2 road(s) is(are) generating traffic...

Road 1 got red light. Waiting for a green light...
1 road(s) is(are) generating traffic...
Road 3 got green light for the next 10 seconds. Generating traffic...
2 road(s) is(are) generating traffic...

Road 2 got red light. Waiting for a green light...
1 road(s) is(are) generating traffic...
Road 1 got green light for the next 10 seconds. Generating traffic...
2 road(s) is(are) generating traffic...

Road 3 got red light. Waiting for a green light...
1 road(s) is(are) generating traffic...
Road 2 got green light for the next 10 seconds. Generating traffic...
2 road(s) is(are) generating traffic...

Road 1 got red light. Waiting for a green light...
Road 3 got green light for the next 10 seconds. Generating traffic...
1 road(s) is(are) generating traffic...2 road(s) is(are) generating traffic...


Road 2 got red light. Waiting for a green light...
1 road(s) is(are) generating traffic...
Road 1 got green light for the next 10 seconds. Generating traffic...
2 road(s) is(are) generating traffic...

Road 3 got red light. Waiting for a green light...
1 road(s) is(are) generating traffic...
Road 2 got green light for the next 10 seconds. Generating traffic...
2 road(s) is(are) generating traffic...

Road 1 got red light. Waiting for a green light...
Road 3 got green light for the next 10 seconds. Generating traffic...
1 road(s) is(are) generating traffic...
2 road(s) is(are) generating traffic...

Road 2 got red light. Waiting for a green light...
1 road(s) is(are) generating traffic...
Road 1 got green light for the next 10 seconds. Generating traffic...
2 road(s) is(are) generating traffic...

Road 3 got red light. Waiting for a green light...
1 road(s) is(are) generating traffic...
Road 2 got green light for the next 10 seconds. Generating traffic...
2 road(s) is(are) generating traffic...
Closing road 3 ...
Road 3 got request to get closed
Road 3 closed
Closing road 2 ...

Road 1 got red light. Waiting for a green light...
1 road(s) is(are) generating traffic...
Road 1 got green light for the next 10 seconds. Generating traffic...
2 road(s) is(are) generating traffic...

Road 2 got red light. Waiting for a green light...
1 road(s) is(are) generating traffic...
Road 2 got request to get closed
Road 2 closed
Closing road 1 ...

Road 1 got red light. Waiting for a green light...
0 road(s) is(are) generating traffic...
Road 1 got request to get closed
Road 1 closed

The example above shows how semaphore controls access to the resource by multiple threads. But please note that resource was NOT made thread safe! Semaphore was just allowing up to N threads (2 in our case) to be active at a time (and generate the traffic towards the bridge). If we wanted to limit the number of vehicles on the bridge and to control traffic lights depending on the current bridge load, we would have had to limit the number of active threads to 1. In that case only one road would have green light at a time.

In the next example I want to show how to synchronise multiple processes in accessing shared resource. Let's say we have an app which writes some log into the file and does it in a loop. The code could look like this:

main.cpp:



If run with parameter of value e.g. "012345", this app will create text file with the following content:

test.log:

[PID = 4408] Iteration # 1 012345
[PID = 4408] Iteration # 2 012345
[PID = 4408] Iteration # 3 012345
[PID = 4408] Iteration # 4 012345
...

If we run simultaneously two or more instances of this process, they will all write into the same file, increasing its size with each write operation. Manipulator endl inserts new line character ('\n') at the end of the line and flushes the buffer to the disk. Obviously, before writing to the disk, our file stream object needs to know the current size of the file in order to move write pointer to the file end. If Process2 appends a new line to the file after Process1 reads file size but before Process1 writes into it, Process2 will increase file size but Process1 will know only about the previous file size and start writing at the position set accordingly, effectively overwriting Process2's last written line!

The following code is the content of the script (DOS batch file) which runs three instances of our application, providing each with different argument:

run_processes.bat:

@start Process.exe 012345
@start Process.exe ABCDEFHIJKLM
@start Process.exe 987654321

Arguments are of different length so we can easily detect the place of the corruption in the output file, like this one:

test.log:

...
[PID = 8280] Iteration # 214 ABCDEFHIJKLM
[PID = 8120] Iteration # 208 987654321
M
[PID = 8280] Iteration # 216 ABCDEFHIJKLM
...

What happened here? First of all, we need to know that on Windows, \n read from stream buffer is expanded to \r\n (CR-LF) before writing it to the file on disk. Process 8120 has updated its knowledge of file size but before it wrote iteration #208 log, process 8280 had written its log for iteration #215 so basically we had this:

test.log (showing hidden CR-LF characters):

...
[PID = 8280] Iteration # 214 ABCDEFHIJKLM\r\n
[PID = 8280] Iteration # 215 ABCDEFHIJKLM\r\n
...

Then process 8120 wrote its #208 log, but effectively overwriting 8280's #215, after what 8280 wrote its log #216:

test.log (showing hidden CR-LF characters):

...
[PID = 8280] Iteration # 214 ABCDEFHIJKLM\r\n
[PID = 8120] Iteration # 208 987654321\r\nM\r\n
[PID = 8280] Iteration # 216 ABCDEFHIJKLM\r\n
...

Obviosuly, we need to protect file so only one process is accessing it at a time. We can do that with semaphore or mutex which are shared between multiple processes (and therefore must be named).

In this article I will show how to achieve it with semaphore:

main.cpp:



All processes are competing to get semaphore signal. First process whose wait() returns (decreasing semaphore count by 1 - possibly to minimal value of 0 in which case all other processes block on their wait()) gets a exclusive access to a file and updates its content after which it releases semaphore (increasing its count to 1 again). All processes are competing again and the one whose wait() returns first gets its slot of exclusive access. There is no corruption in the file any more:

test.log:

...
[PID = 6036] Iteration # 213 987654321
[PID = 10940] Iteration # 213 012345
[PID = 11644] Iteration # 213 ABCDEFHIJKLM
[PID = 6036] Iteration # 214 987654321
[PID = 10940] Iteration # 214 012345
[PID = 11644] Iteration # 214 ABCDEFHIJKLM
[PID = 6036] Iteration # 215 987654321
[PID = 10940] Iteration # 215 012345
[PID = 11644] Iteration # 215 ABCDEFHIJKLM
...

Note: Although this example uses semaphore for process synchronisation, mutex is here more natural solution - we don't want to limit number of accessors to several (N) but only to 1. Only process that locks the mutex can unlock it and with binary semaphore we are just emulating this behaviour.

Links and References:
Semaphore Objects (MSDN)
Using Semaphore Objects (MSDN)
Semaphore (Wikipedia)
Windows Thread Synchronization - Synchronization Using Semaphores
Joseph M. Newcomer: Semaphores
Mutex or Semaphore for Performance?
Mutex vs Semaphore
Mutex vs Semaphores
Difference between binary semaphore and mutex

Friday, 4 November 2011

How to get notified when network adapters' IP addresses change? (Windows OS)

TCP/IP networking software requires at least one network adapter bounded to TCP/IP protocol stack. Each physical network card has unique MAC address and static or dynamic IP address. Static address (one or possibly more) is configured manually and dynamic address is assigned by Dynamic Host Configuration Protocol (DHCP) server (if DHCP is enabled; Obtain IP address automatically is ticked in TCP/IPv4 Properties window). Operating system maintains table which maps IP addresses to network interfaces. Use ipconfig command in command prompt window to display that table. If your adapter has DHCP enabled and you unplug/plug network cable (if using LAN adapter), disable/enable network adapter through adapter settings, issue ipconfig /release or /renew command, get out of/into WiFi range (if using WiFi adapter)...your adapter will loose existing or get a new IP address. NIC to IP address table will change.

Networking applications often need to be aware when changes in this table occur. Windows API function NotifyAddrChange notifies caller on this event. Here is an example of how to call it synchronously (when it's blocking):

#if !defined(_MT)
#error _beginthreadex requires a multithreaded C run-time library.
#endif

#include <winsock2.h>
#include <iphlpapi.h>
#include <Windows.h>
#include <iostream>
#include <sstream>
#include <process.h>
#include <iomanip>

#pragma comment(lib, "iphlpapi.lib")
#pragma comment(lib, "ws2_32.lib")

#define APP_RETURN_CODE_SUCCESS 0
#define APP_RETURN_CODE_ERROR 1

#define THREAD_RETURN_CODE_SUCCESS 0
#define THREAD_RETURN_CODE_ERROR 1

// 127.0.0.1 in network byte order
#define IP_LOCALHOST 0x0100007F

void PrintIPTable()
    PMIB_IPADDRTABLE pIPAddrTable;

 pIPAddrTable = (MIB_IPADDRTABLE*)malloc(sizeof(MIB_IPADDRTABLE));

 if(!pIPAddrTable) 
 {
  std::cout << "malloc() failed" << std::endl;
  return;
 }
  
 // Before calling AddIPAddress we use GetIpAddrTable to get
 // an adapter to which we can add the IP
 // Make an initial call to GetIpAddrTable to get the
 // necessary size into the dwSize variable
 
 DWORD dwSize = 0;

 if(GetIpAddrTable(pIPAddrTable, &dwSize, 0) == ERROR_INSUFFICIENT_BUFFER) 
 {
  free(pIPAddrTable);
  pIPAddrTable = (MIB_IPADDRTABLE*)malloc(dwSize);

  if(!pIPAddrTable) 
  {
   std::cout << "malloc() failed" << std::endl;
   return;
  }
 }
 
 DWORD dwRetVal = 0;
 LPVOID lpMsgBuf;

 // Make a second call to GetIpAddrTable to get the actual data we want
 if((dwRetVal = GetIpAddrTable(pIPAddrTable, &dwSize, 0)) != NO_ERROR ) 
 { 
  std::cout << "GetIpAddrTable() failed. Error code: " << dwRetVal << std::endl;
  
  if(FormatMessage(
   FORMAT_MESSAGE_ALLOCATE_BUFFER | FORMAT_MESSAGE_FROM_SYSTEM | FORMAT_MESSAGE_IGNORE_INSERTS, 
   NULL, dwRetVal, MAKELANGID(LANG_NEUTRAL, SUBLANG_DEFAULT),       
   (LPTSTR) & lpMsgBuf, 0, NULL)) 
  {
   std::cout << "\tError: " << lpMsgBuf << std::endl;
   LocalFree(lpMsgBuf);
  }

  free(pIPAddrTable);
  return;  
 }

 std::cout << "\tNum Entries: " << pIPAddrTable->dwNumEntries << std::endl;
  
 for(int i = 0; i < (int)pIPAddrTable->dwNumEntries; i++) 
 {   
  std::cout << "\n\tInterface Index[" << i << "]:\t" << pIPAddrTable->table[i].dwIndex << std::endl;

  IN_ADDR IPAddr;

  IPAddr.S_un.S_addr = (u_long) pIPAddrTable->table[i].dwAddr;
  std::cout << "\tIP Address[" << i << "]:     \t" << inet_ntoa(IPAddr) << std::endl;

  IPAddr.S_un.S_addr = (u_long) pIPAddrTable->table[i].dwMask;
  std::cout << "\tSubnet Mask[" << i << "]:    \t" << inet_ntoa(IPAddr) << std::endl;

  IPAddr.S_un.S_addr = (u_long) pIPAddrTable->table[i].dwBCastAddr;
  std::cout << "\tBroadCast[" << i << "]:      \t" << inet_ntoa(IPAddr) << "(" << pIPAddrTable->table[i].dwBCastAddr << ")" << std::endl;
  std::cout << "\tReassembly size[" << i << "]:\t" << pIPAddrTable->table[i].dwReasmSize << std::endl;

  std::cout << "\tType and State[" << i << "]:";

  if(pIPAddrTable->table[i].wType & MIB_IPADDR_PRIMARY)
   std::cout << "\tPrimary IP Address";

  if(pIPAddrTable->table[i].wType & MIB_IPADDR_DYNAMIC)
   std::cout << "\tDynamic IP Address";

  if(pIPAddrTable->table[i].wType & MIB_IPADDR_DISCONNECTED)
   std::cout << "\tAddress is on disconnected interface";

  if(pIPAddrTable->table[i].wType & MIB_IPADDR_DELETED)
   std::cout << "\tAddress is being deleted";

  if(pIPAddrTable->table[i].wType & MIB_IPADDR_TRANSIENT)
   std::cout << "\tTransient address";

  std::cout << std::endl;

  if(IP_LOCALHOST == pIPAddrTable->table[i].dwAddr)
  {
   std::cout << "\tLOCALHOST interface" << std::endl;
   // continue;
  }
    
  MIB_IFROW iInfo;
  memset(&iInfo, 0, sizeof(MIB_IFROW));
  iInfo.dwIndex = pIPAddrTable->table[i].dwIndex;
  GetIfEntry(&iInfo);

  std::cout << "\tNetwork interface name: " << iInfo.bDescr << std::endl;

  std::cout << "\tNetwork interface type: ";

  switch(iInfo.dwType)
  {
  case MIB_IF_TYPE_OTHER:
   std::cout << "OTHER" << std::endl;
   break;
  case MIB_IF_TYPE_ETHERNET:
   std::cout << "ETHERNET" << std::endl;
   break;
  case MIB_IF_TYPE_TOKENRING:
   std::cout << "TOKENRING" << std::endl;
   break;
  case MIB_IF_TYPE_FDDI:
   std::cout << "FDDI" << std::endl;
   break;
  case MIB_IF_TYPE_PPP:
   std::cout << "PPP" << std::endl;
   break;
  case MIB_IF_TYPE_LOOPBACK:
   std::cout << "LOOPBACK" << std::endl;
   break;
  case MIB_IF_TYPE_SLIP:
   std::cout << "SLIP" << std::endl;
   break;
  }

  const int unMACSegmentsCount = 6;

  if(unMACSegmentsCount == iInfo.dwPhysAddrLen)
  {      
   std::ostringstream ossMAC;
   ossMAC.fill('0');
   
   ossMAC << std::setw(2) << std::hex << static_cast<unsigned int>(iInfo.bPhysAddr[0]);

   for(int i = 1; i < unMACSegmentsCount; i++)
   {
    ossMAC << '-' << std::setw(2) << std::hex << static_cast<unsigned int>(iInfo.bPhysAddr[i]);
   }
    
   std::cout << "\tMAC Address:            " << ossMAC.str() << std::endl;   
  }  

  std::cout << std::endl;
 } 

 if (pIPAddrTable) 
 {
  free(pIPAddrTable);
  pIPAddrTable = 0;
 }
}

unsigned __stdcall thfn(void* args)
{
 HANDLE hTerminateEvent = *(HANDLE*)args;
 BOOL bTerminate = FALSE;
 
 OVERLAPPED overlap;
 overlap.hEvent = WSACreateEvent();

 if(overlap.hEvent == WSA_INVALID_EVENT)
 {
  std::cout << "WSACreateEvent() failed. Error code: " << WSAGetLastError() << std::endl;
  return THREAD_RETURN_CODE_ERROR;
 }

 while(!bTerminate)
 { 
  
  HANDLE h = 0;

  // call NotifyAddrChange in synchronous mode
  DWORD dwRetVal = NotifyAddrChange(&h, &overlap);

  if(dwRetVal != ERROR_IO_PENDING)
  {
   std::cout << "NotifyAddrChange() failed. Error code: " << WSAGetLastError() << std::endl;
   break;
  }

  HANDLE waitObjects[2] = {hTerminateEvent, overlap.hEvent};

  std::cout << "\nWaiting for IP Table change or termination request..." << std::endl;

  dwRetVal = WaitForMultipleObjects(2, waitObjects, FALSE, INFINITE);

  switch(dwRetVal)
  {
  case WAIT_OBJECT_0:
   std::cout << "WaitForSingleObject(waitObjects) returned WAIT_OBJECT_0 (hTerminateEvent is signaled)" << std::endl;
   bTerminate = TRUE;
   break;
  case WAIT_OBJECT_0 + 1:
   std::cout << "WaitForSingleObject(waitObjects) returned WAIT_OBJECT_0 + 1 (overlap.hEvent is signaled)" << std::endl;   
   
   if(!WSAResetEvent(overlap.hEvent))
   {
    std::cout << "WSAResetEvent() failed. Error code: " << WSAGetLastError() << std::endl;
    bTerminate = TRUE;
    break;
   }

   PrintIPTable();
   break;
  case WAIT_FAILED:
   std::cout << "WaitForSingleObject(waitObjects) returned WAIT_FAILED (function failed)" << std::endl;
   bTerminate = TRUE;
   break;
  }  
 }
 
 if(!WSACloseEvent(overlap.hEvent))
 {
  std::cout << "WSACloseEvent() failed. Error code: " << WSAGetLastError() << std::endl;
  return THREAD_RETURN_CODE_ERROR;
 }

 return THREAD_RETURN_CODE_SUCCESS;
}

void WaitUserInput()
{
 std::cin.clear(); 
 std::cin.ignore(1, '\n');
}

// NOTE: std::cout is shared between two threads and is not thread safe!
int main(int argc, char* argv[])
{
// std::cout << "main()" << std::endl;

 HANDLE hTerminateEvent = CreateEvent(0, TRUE, FALSE, 0);

 if(!hTerminateEvent)
 {
  std::cout << "CreateEvent() failed. Error code: " << GetLastError() << std::endl;
  return APP_RETURN_CODE_ERROR;
 }

 unsigned unThreadID = 0;
 HANDLE hThread = 0;

// std::cout << "Creating thread..." << std::endl;
 hThread = (HANDLE) _beginthreadex(0, 0, thfn, &hTerminateEvent, 0, &unThreadID);

 if(!hThread)
 {
  std::cout << "_beginthreadex() failed. Error code: " << errno << std::endl;
  return APP_RETURN_CODE_ERROR;
 }

 // (not reliable way to) make sure child thread has started before main thread continues
 Sleep(1000);

// std::cout << "Created thread with ID: " << unThreadID << std::endl;

 std::cout << "Press ENTER to terminate listening for changes in IP address table..." << std::endl;
 WaitUserInput();

 SetEvent(hTerminateEvent);

 // wait for thread to terminate
 DWORD dwRetVal = WaitForSingleObject(hThread, INFINITE);

 switch(dwRetVal)
 {
 case WAIT_OBJECT_0:
  std::cout << "WaitForSingleObject(hThread) returned WAIT_OBJECT_0 (event is signaled)" << std::endl;
  break;
 case WAIT_TIMEOUT:
  std::cout << "WaitForSingleObject(hThread) returned WAIT_TIMEOUT (timeout elapsed; event is nonsignaled)" << std::endl;
  break;
 case WAIT_FAILED:
  std::cout << "WaitForSingleObject(hThread) returned WAIT_FAILED (function failed)" << std::endl;
  break;
 }

 if(!CloseHandle(hThread))
 {
  std::cout << "CloseHandle() failed. Error code: " << GetLastError() << std::endl;
  return APP_RETURN_CODE_ERROR;
 }

 std::cout << "Press ENTER to exit..." << std::endl;
 WaitUserInput();

// std::cout << "~main()" << std::endl;
 return APP_RETURN_CODE_SUCCESS;
}

This is the output if we unplug network cable and then plug it back again:


Waiting for IP Table change or termination request...
Press ENTER to terminate listening for changes in IP address table...
WaitForSingleObject(waitObjects) returned WAIT_OBJECT_0 + 1 (overlap.hEvent is s
ignaled)
Num Entries: 2

Interface Index[0]: 17
IP Address[0]: 192.168.56.1
Subnet Mask[0]: 255.255.255.0
BroadCast[0]: 1.0.0.0(1)
Reassembly size[0]: 65535
Type and State[0]: Primary IP Address
Network interface name: VirtualBox Host-Only Ethernet Adapter
Network interface type: ETHERNET
MAC Address: 08-00-27-00-68-56


Interface Index[1]: 1
IP Address[1]: 127.0.0.1
Subnet Mask[1]: 255.0.0.0
BroadCast[1]: 1.0.0.0(1)
Reassembly size[1]: 65535
Type and State[1]: Primary IP Address
LOCALHOST interface
Network interface name: Software Loopback Interface 1
Network interface type: LOOPBACK


Waiting for IP Table change or termination request...
WaitForSingleObject(waitObjects) returned WAIT_OBJECT_0 + 1 (overlap.hEvent is s
ignaled)
Num Entries: 3

Interface Index[0]: 10
IP Address[0]: 192.168.253.122
Subnet Mask[0]: 255.255.255.0
BroadCast[0]: 1.0.0.0(1)
Reassembly size[0]: 65535
Type and State[0]: Primary IP Address Dynamic IP Address
Network interface name: Intel(R) 82566DC Gigabit Network Connection
Network interface type: ETHERNET
MAC Address: 00-19-d1-1b-e0-88


Interface Index[1]: 17
IP Address[1]: 192.168.56.1
Subnet Mask[1]: 255.255.255.0
BroadCast[1]: 1.0.0.0(1)
Reassembly size[1]: 65535
Type and State[1]: Primary IP Address
Network interface name: VirtualBox Host-Only Ethernet Adapter
Network interface type: ETHERNET
MAC Address: 08-00-27-00-68-56


Interface Index[2]: 1
IP Address[2]: 127.0.0.1
Subnet Mask[2]: 255.0.0.0
BroadCast[2]: 1.0.0.0(1)
Reassembly size[2]: 65535
Type and State[2]: Primary IP Address
LOCALHOST interface
Network interface name: Software Loopback Interface 1
Network interface type: LOOPBACK


Waiting for IP Table change or termination request...

WaitForSingleObject(waitObjects) returned WAIT_OBJECT_0 (hTerminateEvent is sign
aled)
WaitForSingleObject(hThread) returned WAIT_OBJECT_0 (event is signaled)
Press ENTER to exit...