Click here to Skip to main content
65,938 articles
CodeProject is changing. Read more.
Articles / web / Node.js

Web System for Smart Image Viewing: Upload, Deep Zoom, Mark Out and Annotation

5.00/5 (24 votes)
1 Dec 2016CPOL17 min read 34.5K   601  
Software for large image smart viewing, mark out and annotation
Compact software for images upload, processing to tile pyramid and subsequent Deep Zoom viewing with custom mark out and annotation

Table of Contents

Image 1

Figure 1. The system structure.

Introduction

In many modern applications, there are requirements for easy and efficient work with large high resolution images. Users need tools to load and view such images with seamless zooming and pan capabilities without the loading time or latency, mark out and annotate certain regions of them. It's even better to have these tools available as platform independent Web applications with zero footprints on a user's machine / device. To address this need, the technology called Deep Zoom (DZ) [1] was developed by Seadragon Software team [2] within Microsoft. DZ was shipped with Microsoft Silverlight used in Silverlight Multiscale Image control. It worked just fine in Web applications but required browser support for Silverlight [3]. Open Seadragon (OSD) project [4] provides similar capabilities framework agnostic as a completely open source. This work presents Web based software using OSD DZ technology for large images viewing.

OSD provides developer API [5] available from JavaScript (JS) to be used in Web browser. This API also supports overlay mechanism [6] for mark out. OSD overlays use "flying" HTML elements placed over selected regions in image. It seems that such an approach is not quite handy for online mark out, particularly for regions with irregular shape. So, a different technique for mark out and annotation based on direct drawing in canvas rather than HTML elements usage is implemented in this work. The structure of the system is depicted in Figure 1. Components of the presented software are listed in the following table:

Table. Software Components.

Component Description Tools
Image Upload Server Image upload to file storage and start of Image Processor to make image tile pyramid node.js, JavaScript
Upload Page A simple Web page backing by Image Upload Server providing UI for the upload HTML
Image Processor Component to create image tile pyramid out of original image .NET Core DLL (currently running in Windows only)
Image Viewing Server Supports viewing Web application node.js, JavaScript
Viewing Web Application Web application for viewing, mark out and annotation of the image HTML, CSS, JavaScript, TypeScript

Functionality

The system works as follows. When user would like to upload an image, she/he browses on https://UploadServerURL:15005 and gets a very simple page for image upload. Then the user finds the required image and by pressing button Upload, posts the image to Upload Server. The latter accepts uploaded file, places it to a file storage and runs Image Processor in a separate process. Image Processor dzPyramidBuilder.dll is a .NET Core DLL. Its mission is to create image tile pyramid from an original image. When the image pyramid has been created, the image is ready for DZ viewing and annotation performed by the viewing subsystem.

To view and possibly annotate the image, the user should browse on https://ViewingServerURL:15000 yielding the page shown in Figures 2a and 2b.

Image 2

Figure 2a. The Image Viewer in viewing mode.

Image 3

Figure 2b. The Image Viewer in viewing mode (fragment).

The Viewer is initially opened in viewing mode (Figures 2a and 2b). In this mode, the Viewer's toolbar includes dropbox of images is available for viewing, following by Layout and Edit toggles. When the Viewer page was loaded, it requests Viewing Server for the list of images available for viewing. In our code sample, for the sake of simplicity, it was assumed that Upload Server, images tile pyramid file storage and Viewing Server are located in the same machine. So to provide the client with the required list Viewer Server retrieves names of subfolders from a well-known folder containing all available image pyramids. Upon receiving it by the Web Viewer application, the list is placed to the client toolbar dropbox. The user selects an image name from the dropbox and views the image along with its annotation, if any. Image annotation is placed into a side panel which locates either to the left or to the right of the image panel. The panel location may be switched with Layout toggle.

Working in viewing mode, the user can zoom and pan selected images and see already marked out selected regions along with their annotations. Our selected regions differ from native OSD overlays. According to OSD documentation [6], the overlays constitute HTML objects, whereas our selected regions are simply drawn on OSD viewer canvas. This approach seems to be more flexible permitting users to define non-rectangular selected regions and manage them dynamically as it will be shown below. When the user zooms and pans image, selected regions are moving accordingly sticking to the image similarly to OSD overlays. In our sample, selected region can be of three types: rectangular, circular and polygonal. When the user hovers the mouse pointer over the selected region in the image, the dynamic tooltip depicts the selected region title. For convenience of their management, selected regions are grouped to layers. All selected regions belong to the same layer may be shown / hidden either individually or by toggle the entire layer. Also, annotations of selected regions may be expanded / collapsed in similar manner.

By clicking Edit toggle user may switch Viewer from viewing mode to editing. In editing mode, image is frozen in its current state (cannot be moved, zoomed or panned), and the user may edit the image. New instruments appear in both the toolbar (Figure 3) and the annotation table.

Image 4

Image 5

Image 6

Image 7

Figure 3. The Image Viewer toolbar in edit mode (with checkered flag indicating that new images are available).

Now it is possible to either edit an already existing selected region (changing its title, annotation and z-order within its layer) or create a new selected region and possibly a new layer. After editing completion, user may save its result if she/he is satisfied with them. Alternatively, user may undo editing to the point of the last save.

In order to create a new selected region, the user should select its type from Type dropbox (could be Rectangle, Circle or Polygon), choose the frame and background color and opacity with color picker control, define frame thickness (between 0 and 9 pixels) and then actually mark a selected region by mouse clicks on the image. In the case of rectangle and circle, the selected region is defined by just two clicks (diagonal points for rectangle and center and a point defining radius for the circle). A polygon user may define as many points as she/he wants and at the end close polygon pressing appropriate button at the toolbox. After selected region shape has been created, modal window to define its text information appears (depicted in Figure 4; the same window is used for editing of already existed selected region after appropriate toggle is clicked against required entry at annotation table). With this modal window, a layer (either existed or newly created) may be assigned to the selected region.

Image 8

Figure 4. Modal window to edit selected region.

The annotation table contains visual tools (Figure 5) to show / hide a selected region or the entire layer, change z-order of a selected region within its layer, edit text information (by opening edit modal window), or just delete selected region (layer also may be deleted if it does not contain any selected region).

Image 9

Figure 5. Editing tools in Annotation table.

And now when we are familiar with functionality, let's briefly look at the code.

Code Description

The structure of the software is depicted in the flow chart at the beginning of the article (Figure 1). It consists of image upload and image viewing subsystems.

Upload

The upload subsystem consists of the Upload Server dzImageUpload, simple Web page ./dzImageUpload/index.html and a processor dzImagePyramidBuilder for building DZ image pyramid. dzImageUpload Web server is written in JS and running by node.js with express and multer packages. The server listens on port 15005 and supports HTTPS GET and POST requests. When user browses URL https://UploadServerURL:15005 Upload Server responds with a simple page ./dzImageUpload/index.html for the upload. With this page, user selects image file and actually uploads it to the server. The upload is performed with HTTPS POST. The server's POST handler carries out the following tasks:

  • Reload ./dzImageUpload/index.html page to browser for the next upload,
  • Save uploaded file to disk using multer package,
  • Run dzImagePyramidBuilder to prepare tile pyramid for the uploaded image (this operation is currently available only in Windows OS - please see explanations below), and finally
  • Notify Viewing Server dzImageView that a new image is ready for viewing and annotation.

For this matter, dzImageUpload acts as a client of dzImageView using function:

JavaScript
var req = https.request(options, function (res) {
               //.....
          });

with parameters:

JavaScript
var agentOptions = {
    host: 'localhost',
    port: 15000,
    path: '/',
    rejectUnauthorized: false
};

var options = {
    url: 'https://localhost:15000',
    agent: new https.Agent(agentOptions),
    method: 'GET'
};

Because we assume that all components of our sample are running in the same machine, localhost IP address may be used for inter-server communication.

The dzImagePyramidBuilder component is an image pyramid builder adopted from [3] and converted to .NET Core. In our sample, it is used as a black box and therefore its description is out of scope of this article. Explanations regarding this component structure may be found in [3] and algorithm used explained in [1] and works on DZ imaging. The component was converted to .NET Core in order to make it compatible with different OS. But this became problematic. The issue is that dzImagePyramidBuilder uses System.Drawing assembly which is not currently supported by .NET Core. So the third party package CoreCompat.System.Drawing [7]. was chosen for replacement. It works fine in Windows but being deployed to Linux failed due to lack of Linux ready made support for GDI+ internally used by the library. So although the image upload works fine in Linux, pyramid preparation from the uploaded files is not available in our sample under Linux. Because the main focus of this article is DZ image viewing and annotation, I allowed myself not to dig deeper for GDI+ support in Linux (your suggestions will be most welcome!) Also OSD recommends several Linux compatible image pyramid builders in its dedicated page [8]. I believe that Microsoft will provide support for its System.Drawing assembly and particularly for Bitmap type in upcoming releases of .NET Core.

Viewing, Mark Out and Annotation

Viewing subsystem is the main part of the code sample. It consists of dzImageView Web server written in JS for node.js with express package and ./dzImageView/dzweb Web single page application (SPA) written in JS and partly in TypeScript (TS).

dzImageView Web server listens on port 15000. When user browses URL https://ViewingServerURL:15000, the server sends back ./dzImageView/dzweb/index.html page starting dzweb Web application. The application may run with two parameters, namely:

  • lang=xx - language of UI, where "xx" stands for the two leading letters of the language, e.g., "en" for English (it is default), and
  • layout=x - denotes mutual placement of image and annotation panels. By default, the annotation panel is placed to the right from the image panel. And if layout parameter is set fore either "l" or "L", then annotation panel is placed to the left from the image panel.

So the URL:

HTML
https://ViewingServerURL:15000?lang=ru&layout=l

causes the viewer with UI in Russian language with annotation at the left-hand side of the browser window.

dzweb is developed using mostly plain JS and CSS. Standard and the third party packages are used only for WebSocket communication (socket.io), selected region color picker (jcolor) and of course, for DZ itself (openseadragon). TS is used for selected region types described in the following files in the ./dzImageView/dzweb/ts folder: base type in selectedRegion.ts and derived types in selectedRegionCircle.ts, selectedRegionPolygon.ts and selectedRegionRectangle.ts. TS files should be transpiled to JS offline as a part of system preparation. This can be done with tsc.exe transpiler (has to be installed) by the following command from ./dzImageView/dzweb/ts folder:

"C:\Program Files (x86)\Microsoft SDKs\TypeScript\2.0"\tsc.exe

The command uses the tsconfig.json file. For the user's convenience, the above command is placed into command file tsc.cmd. Its output according to parameter "compilerOptions"."outDir" will be put to the ./dzImageView/dzweb/js/selectedRegion folder.

The bulk of our Web application is located in the ./dzImageView/dzweb/js folder. We are already familiar with the selectedRegion subfolder containing results files of TS transpiling. The dzweb application is designed with strict separation between HTML elements and business logic. Files dealing with HTML elements are placed to separate page subfolder and named page*.js. Files with code free of HTML tags located in the ./dzImageView/dzweb/js folder. Page*() functions are called by non-page code directly. Calls in opposite direction are performed with appropriate callback provided to page* code.

Let's have a brief look at dzweb code. Code execution starts with function startUp() from file global.js. It is called from HTML page with:

JavaScript
<script src="js/mainDz.js" onload="startUp()"></script>

Function startUp() activates theSingleton object (file singleton.js). The theSingleton object is used to hold variables used across the application thus avoiding global variables. In startUp() theSingleton starts WebSocket communication with server and requests information about available images with 'init' message. Communication with server via WebSocket is implemented in function exchangeWithServer() using Promise for asynchronous operations. It takes as parameters message name, object to send to server and a callback function to process server's response. When Web application received list of available images from server, it creates PageInfo object (file ./dzImageView/dzweb/js/page/page.js) with new method passing list of images and callback getImageInfo() for getting detailed data of an image as parameters to PageInit(). When details for the first image have been received from server, dzweb completes its initialization and shows the first image along with its mark out and annotation, if available. Below main functionality is described by files.

  • global.js provides global functions for the entire application. It also includes constants:
    JavaScript
    const hostIPAddress = '$0';
    const __isRightLayout = $1;

    which dollar-leading placeholders are replaced by server with actual values taken from client request. This is important particularly for hostIPAddress parameter to provide Web application with a proper IP address of the server.

  • mainDz.js contains most of OSD related code and WebSocket message handler.
  • layerCollection.js and layer.js contain LayerCollection and Layer objects respectively handling layers and selected regions collections.
  • transform.js is for Transform object responsible for web to image and vice versa coordinates transform.
  • tip.js contains Tip object for drawing tooltips when mouse moves over image's selected regions.
  • There are some features that are worth to mention.

Sticking Selected Region to Image

Moving selected regions along with image ("sticking") is achieved mainly with two pieces of code. The following handler is attached to OSD viewer object update-viewport event (file mainDz.js).

JavaScript
viewer.addHandler('update-viewport', function () {
    loopThroughAllSelectedRegions(function (selectedRegion) {
        selectedRegion.draw(viewer);
    });
});

Function selectedRegion.draw(viewer) is implemented by base type SelectedRegion. This function calls function public drawInternal(vr: any, ctx: any) implemented by derived selected region types. drawInternal() includes the following point coordinates transform:

JavaScript
Transform.prototype.imageToWeb = function (viewer, imagePoint) {
    var vpt = viewer.viewport;
    var ptWeb = vpt.pixelFromPoint(vpt.imageToViewportCoordinates(imagePoint), 
		    	    true); //!
			
    return osdPoint(ptWeb.x, ptWeb.y);
}

The second argument of function pixelFromPoint of viewer.viewport should be set to true indicating current point's location.

Iconic Font Awesome

This glyph font [9] is used to decorate visual tools, particularly buttons. In the sample, only fontawesome-webfont.ttf font file is used. Font files with extensions woff and woff2 were removed from installation, and Viewing Server reports error on attempt to download them. These error messages may be ignored.

Localization

As it was stated above, user may specify parameter lang=xx defining UI language. Viewing Server downloads file ./dzImageView/dzweb/settings/settings_xx.js as part of Viewing Web application. By default, English language file is used. For simplicity, the setting file currently contains all settings expected by the Web application (e.g., colors for tooltips). Currently, the Russian language setting file is available for test along with the English one.

Mutually Exclusive Editing

When the user starts to edit a particular image, all other users are not allowed to edit this image (toggle Edit disappears from toolbar for this image) until the user who "locked" the image will abandon edit mode for the image. If the user-editor saved editing results, then server updates clients who currently view the image. It is possible that user-editor closes Web application being in edit mode. This can "lock" editing image until server restart. To avoid this problem and "unlock" the image permitting other users to edit it, the Viewing Server periodically (the period is defined by constant socketsCheckTimeoutInSec and currently set to 10 seconds) and on event of a new client connection, checks validity of all clients connections, removing invalid ones and unlocks images edited by unconnected clients. Please note that this is only the sample, and a real world system should have some permission mechanism for editing and update.

New Image Availability Notification

When new image has been processed to a tile pyramid and become ready to viewing, the server sends notification to all running clients. The checkered flag sign at the right of toolbar indicates that new image is available and will be included to the image list on the next reload of Web page.

Running Sample

Node.js should be installed to run both upload and viewing servers.

Windows

In order to run upload subsystem .NET Core should be installed to run dzImagePyramidBuilder component. The sample was tested with .NET Core version 1.0.0. Viewing subsystem requires no prerequisites. Course of action is following:

  1. Unzip files to some root folder.
  2. Go to folder ./dzImagePyramidBuilder/src/dzImagePyramidBuilder and run file publish.cmd containing the following command:
    dotnet publish -o ../../../dzImageUpload/pyramidBuilder -c Release

    It will build dzImagePyramidBuilder and publish it to ./dzImageUpload/pyramidBuilder folder.

  3. Optional step. Go to folder ./dzNodeServer/dzweb/ts and run tsc.cmd command file containing command:
    "C:/Program Files (x86)/Microsoft SDKs/TypeScript/2.0"/tsc.exe

    to transpile TS files from this folder to ./dzNodeServer/dzweb/js/selectedRegion folder. Actually, these JS files are already in place.

    Note: Please do not forget to transpile TS files if you will change them.

  4. Run Upload.cmd command file from the root folder. It starts dzImageUpload server with:
    node server

    command from ./dzImageUpload folder.

    Note: The above command runs upload and viewing servers in both Windows and Linux environments.

  5. Open Web Browser (Google Chrome is recommended) and browse URL https://localhost:15005 and upload image file of your choice. The image will be placed to ./dzImageUpload/uploads folder replacing blanks with underscores and adding time in ticks to its name. Image tile pyramid will be output to appropriate subfolder under ./dzImageView/dzweb/images.
  6. Run View.cmd command file from the root folder. It starts dzImageView server from ./dzImageView folder.
  7. Browse URL https://localhost:15000 [optionally extended with: ?lang=en&layout=l] where en may be replaced with ru for the Russian language UI or file prepared by you in your favorite language. Viewer Web application screen depicted in Figures 2a and 2b will appear showing your uploaded image. For editing, click on Edit toggle. After your editing is completed, save its results by pressing Save button and leave edit mode clicking Edit toggle again.

    Debugging the servers and pyramid builder may be carried out opening ./dz.sln solution with Microsoft Visual Studio 2015.

Linux

As it was said above, pyramid builder in its current version can not run in Linux. So even though image upload works fine, there is no reason to run upload subsystem in Linux. Thus only view subsystem is discussed. Version Ubuntu 16.04 is used.

Here, I describe the steps necessary to perform after installation in Windows described in previous section.

  1. Copy Windows root folder to appropriate root folder in Linux (only subfolder dzImageView should be actually copied to the root folder).
  2. Copy already prepared in Windows image pyramids to ./dzImageView/dzweb/images folder.
  3. Run dzImageView server.
  4. View / Edit images as it is described in point 7 for Windows installation.

Further Development

As I have already written above, this software cannot be viewed as a full blown product. So bugs of any sort are more than likely (e.g., circular selected region currently may occupy space outside the image). And of course, there is endless room for UI improvements. Selected region with smooth boundary (perhaps Bezier curve) may be added. Developer API will be useful particularly to assign some custom handlers to events related to selected regions, like click, hover, etc. Current version successfully works with Google Chrome and Mozilla Firefox. In Microsoft browsers (Internet Explorer and Edge), tooltips mechanism should be fixed. Layout may be adjusted to mobile devices.

Conclusion

This work presents software for large image smart viewing, mark out and annotation. Deep Zoom technology is used for viewing large images with high resolution without delay and latency. Upload and Viewing Servers were developed in Node.js with JavaScript being therefore platform independent. Deep Zoom technology implies usage of image tile pyramid. To prepare those pyramids, separate component is used (its current version works only in Windows due to lack of GDI+ ready made support in Linux). Image Viewing Web application was developed with HTML/CSS, JavaScript and TypeScript with minimum third party packages.

References

History

  • 1st December, 2016: Initial version

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)