My Tech Notes: mediaflux

Showing posts with label mediaflux. Show all posts

Mediaflux: service to resolve DNS SRV record

ldap

server.dns.lookup :type SRV :target _ldap._tcp.domain.com

Kerberos

server.dns.lookup :type SRV :target _kerberos._tcp.domain.com

Enable TLS/SSL encrypted DICOM transfer to Mediaflux server (using DCMTK storescu)

I. TLS/SSL encrypted DICOM transfer to Mediaflux DICOM server without verifying the certificates on both ends

Configure Mediaflux server with SSL encryption without client certificate verification, i.e. allow untrusted.

network.start :type dicom :port 6667 :ssl -trusted false true \
    :arg -name asset.engine pss \
    :arg -name authentication.domain dicom \
    :arg -name dicom.title MFLUX \
    :arg -name pss.asset.namespace.root /dicom \
    :arg -name pss.id.subject.by patient.id

Send DICOM files to Mediaflux server using DCMTK's storescu

storescu +tls /path/to/dcmtk-storescu.key /path/to/dcmtk-storescu.pem -ic -aet DCMTK -aec MFLUX mediaflux.example.org 6667 /path/to/*.dcm

NOTE: -ic option in the command above disables peer certificate verification.

II. TLS/SSL encrypted DICOM transfer to Mediaflux DICOM server with certificate verification on both ends

Export Mediaflux server certificate:
```
server.certificate.identity.export :format base64 :id 8 :out file:/path/to/dcmtk-trusted-certs/mflux.pem
```
Note: Currently, there is an issue when the Mediaflux server certificate identity contains chained root CA certificate and intermediate certificate: The server certificate file exported by server.certificate.identity.export service is incomplete. It contains only the host certificate and misses the identity contains also chained root CA certificate and intermediate certificate. This causes the DICOM client using this certificate file failed to connect to Mediaflux DICOM server while doing SSL handshake.

To work around this issue, use the original certificate file that includes all the chained certificates.
Import (DCMTK) client's certificate into Mediaflux as trusted certificate:
```
server.certificate.trust.import :in file:/path/to/dcmtk-storescu.pem
```

Configure Mediaflux server with SSL encryption with certificate verification:

network.start :type dicom :port 6667 :ssl -trusted true true \
    :arg -name asset.engine pss \
    :arg -name authentication.domain dicom \
    :arg -name dicom.title MFLUX \
    :arg -name pss.asset.namespace.root /dicom \
    :arg -name pss.id.subject.by patient.id

Send DICOM files to Mediaflux server using DCMTK's storescu

storescu +tls /path/to/dcmtk-storescu.key /path/to/dcmtk-storescu.pem --add-cert-file /path/to/dcmtk-trusted-certs/mflux.pem --rc -aet DCMTK -aec MFLUX mediaflux.example.org 6667 /path/to/*.dcm

NOTE: -rc option in the command above requires peer certificate verification.
--add-cert-dir specifies the trusted certificate.

Mediaflux aterm xvalues: how to handle values contains spaces

For example, I want to loop through the dictionary entries below:

> dictionary.entries.list :dictionary daris:pssd.study.types
    :term "Bone Densitometry (ultrasound)"
    :term "Cardiac Electrophysiology"
    :term "Combined Results"
    :term "Computed Radiography"
    :term "Computed Tomography"
    :term "Dose Report"
    :term "Electrocardiography"
    :term "Electron Microscopy"
    :term "General Microscopy"
    :term "Intravascular Optical Coherence Tomography"
    :term "Magnetic Resonance Imaging"
    :term "Mammography"
    :term "Nuclear Medicine"
    :term "Optical Microscopy"
    :term "Positron Emission Tomography"
    :term "Positron Emission Tomography/Computed Tomography"
    :term "Quality Assurance"
    :term "Radio Fluoroscopy"
    :term "Slide Microscopy"
    :term "Ultra Sound"
    :term "Unspecified"
    :term "X-Ray Angiography"

The code below CANNOT parse the values properly, because it uses space as the separator:

foreach term [xvalues term [dictionary.entries.list :dictionary daris:pssd.study.types]] {
    puts $term
}

You need to use a different separator char, TCL split function to rebuild the list:

foreach term [split [xvalues term [dictionary.entries.list :dictionary daris:pssd.study.types] ,] ,] {
    puts $term
}

Install SSL certificate on Mediaflux server

Generate certificate request and private key. (The private key can be reused to generate new requests for renewing the certificate.)

Create a request configuration file named your-domain-name.csr.conf like below:

[req]
prompt=no
default_bits=2048
encrypt_key=no
default_md=sha1
distinguished_name=dn
# PrintableStrings only
string_mask=MASK:0002
x509_extensions=x509_ext
req_extensions=req_ext
[dn]
C=AU
ST=Your State
L=Your City
O=Your Organization
OU=Your Organisation Unit
CN=mediaflux.your-domain.org
[x509_ext]
subjectAltName=DNS:name1.your-domain.org.au,URI:https://mediaflux.your-domain.org.au/1234/shibboleth,DNS:name2.your-domain.org.au,URI:https://daris.your-domain.org.au/1234/shibboleth
subjectKeyIdentifier=hash
[req_ext]
subjectAltName=DNS:name1.your-domain.org.au,DNS:name2.your-domain.org.au

Generate certificate request using the command below:
```
openssl req -config your-domain-name.csr.conf -new -days 3650 -keyout your-domain-name.key -out your-domain-name.csr
```
You should now have the generated private key file: your-domain-name.key and request file: your-domain-name.csr. Keep the private key file in safe for future certificate renewal requests.

Submit the generated your-domain-name.csr file to be signed by CA (You only need to sumit the csr file.) And you should get CA signed certificate: your-domain-name.crt
Install the CA signed certificate:

Convert private key to .p8 format:
```
openssl pkcs8 -topk8 -inform PEM -outform DER -nocrypt -in your-domain-name.key -out your-domain-name.key.p8
```
You only need to do this once and keep the your-domain-name.key.p8 file together with your-domain-name.key for future certificate installations.
Install the CA signed certificate using the command below in Aterm:
```
server.certificate.identity.import :in file:/path/to/your-domain-name.key.p8 :in file:/path/to/your-domain-name.crt :replacement true
```
It should return the imported certificate identity entry. Remember the id of the identity to be used in the next step.
Set default certificate alias the the newly installed certificate identity (assume its id is 2):
```
server.property.set :property -name server.default.certificate.alias 2
```

Mediaflux TCL script: loop through query results with cursors

set size 100
set idx 1
set remaining 1

while { $remaining > 0 } {
    set r [asset.query :size $size :idx $idx :count true :where "namespace>=/test"]
    foreach id [xvalues id $r] {
         puts $id
    }
    set idx [expr { $idx + $size }]
    set remaining [xvalue cursor/remaining $r]
}

Mediaflux example: tcl script to repair metadata with empty string as attribute

foreach id [xvalues id [asset.query :where xpath(daris:pssd-derivation/input/@vid)='' :size infinity]] {

    set args ":id ${id} :meta -action remove < :daris:pssd-derivation > :meta < :daris:pssd-derivation"

    set doc [xelement asset/meta/daris:pssd-derivation [asset.get :id ${id}]]

    set doc_id [xvalue daris:pssd-derivation/@id ${doc}]

    set args "$args -id ${doc_id} < "

    set processed [xvalue daris:pssd-derivation/processed ${doc}]

    set args "$args :processed ${processed}"

    foreach input [xvalues daris:pssd-derivation/input ${doc}] {

        set vid [xvalue asset/@vid [asset.get :cid ${input}]]

        set args "$args :input -vid ${vid} ${input}"

    }

    set method [xvalue daris:pssd-derivation/method ${doc}]

    set step [xvalue daris:pssd-derivation/method/@step ${doc}]

    set args "$args :method -step ${step} ${method}"

    set args "$args > >"

    puts "asset.set $args"
 
    asset.set $args

}

JavaCC Template to generate Mediaflux asset path

Mediaflux Plugin Development: Create dataset from HTTP URL (with Basic Auth)

The example shows how to create a dataset asset from a password protected URL (Basic HTTP Auth).

Java: HTTP GET data from URL (with basic HTTP Auth)

The example shows how to do HTTP GET to download a file from URL. It also include an example to call Mediaflux client api to create a dataset from the url by piping the streams.

Multi-threading in Mediaflux Plugin Service

package wxyz.mf.plugin.services;

import java.util.ArrayList;
import java.util.Collection;
import java.util.List;
import java.util.Map;

import arc.mf.plugin.PluginLog;
import arc.mf.plugin.ServiceExecutor;
import arc.xml.XmlDoc.Element;
import arc.xml.XmlWriter;
import wxyz.mf.plugin.util.MultiThreadPluginService;

public class SvcTest extends MultiThreadPluginService<Integer> {

    private Interface _defn;

    public SvcTest() {
        _defn = new Interface();
    }

    @Override
    public Access access() {
        return ACCESS_ACCESS;
    }

    @Override
    public Interface definition() {
        return _defn;
    }

    @Override
    public String description() {

        return "Test multithreading.";
    }

    @Override
    public String name() {
        return "omics.test";
    }

    @Override
    protected void consume(ServiceExecutor executor, Integer input, PluginLog log) throws Throwable {
        Thread.sleep(1000);
        String uuid = executor.execute("server.uuid").value("uuid");
        if (input == 15) {
            log.add(PluginLog.WARNING, "#8: Mock Exception for value 15.");
            throw new Exception("Mock Exception for value 15.");
        }
        System.out.println(Thread.currentThread().getName() + " consumed: " + input);
        System.out.println(Thread.currentThread().getName() + " uuid: " + uuid);
    }

    @Override
    protected Collection<Integer> produce(Element args, Inputs in, Outputs out, XmlWriter w,
            Map<String, Object> intermediateResults) throws Throwable {
        List<Integer> inputs = new ArrayList<Integer>(100);
        for (int i = 0; i < 20; i++) {
            inputs.add(i);
        }
        return inputs;
    }

}

Modify Mediaflux mflux package to install it into non-default schema

copy the mflux.zip from /opt/mediaflux/ext/packages/ to /tmp

cp /opt/mediaflux/ext/packages/mflux.zip /tmp/

mkdir /tmp/mfpkg; cd /tmp/mfpkg; unzip /tmp/mflux.zip

edit __install.tcl file

vi /tmp/mfpkg/__install.tcl

# HTTP processors:

if { [info exists host] } {
    
    if { [xvalue exists [http.processor.exists :host $host :url /mflux]] == "false" } { 
      http.processor.create :host $host :app mflux :url /mflux :type asset :translate /www :authentication < :domain $domain :user $user > :view mflux-www
    }

    # Set the servlet entry points for the HTTP processors.
    http.servlets.set :host $host :url /mflux \
       :servlet -path portal.mfjp -default true arc.mflux.main \
       :servlet -path icon.mfjp arc.mflux.icon \
       :servlet -path iptile.mfjp arc.mflux.image.pyramid.tile \
       :servlet -path meta.mfjp arc.mflux.meta \
       :servlet -path content.mfjp arc.mflux.content \
       :servlet -path share.mfjp arc.mflux.share \
       :servlet -path output.mfjp arc.mflux.session.output \
       :servlet -path execute.mfjp arc.mflux.execute \
       :servlet -path transcode.mfjp arc.mflux.transcode \
       :servlet -path aterm.jar arc.mflux.aterm \
       :servlet -path aar.jar arc.mflux.aar
       
} else {

    if { [xvalue exists [http.processor.exists :url /mflux]] == "false" } { 
      http.processor.create :app mflux :url /mflux :type asset :translate /www :authentication < :domain $domain :user $user > :view mflux-www
    }

    # Set the servlet entry points for the HTTP processors.
    http.servlets.set :url /mflux \
       :servlet -path portal.mfjp -default true arc.mflux.main \
       :servlet -path icon.mfjp arc.mflux.icon \
       :servlet -path iptile.mfjp arc.mflux.image.pyramid.tile \
       :servlet -path meta.mfjp arc.mflux.meta \
       :servlet -path content.mfjp arc.mflux.content \
       :servlet -path share.mfjp arc.mflux.share \
       :servlet -path output.mfjp arc.mflux.session.output \
       :servlet -path execute.mfjp arc.mflux.execute \
       :servlet -path transcode.mfjp arc.mflux.transcode \
       :servlet -path aterm.jar arc.mflux.aterm \
       :servlet -path aar.jar arc.mflux.aar

}

Re-package:

cd /tmp/mfpkg; zip -r ../mflux-1.zip ./*

Install the modified package in Aterm:

package.install :in file:/tmp/mflux-1.zip :arg -name host daris.vicnode.org.au

Mediaflux startup script to run the server as non-root user

The configuration file should be saved to /etc/mediaflux:

export MFLUX_HOME=/opt/mediaflux
export MFLUX_DOMAIN=system
export MFLUX_USER=manager
export MFLUX_TRANSPORT=https
export MFLUX_PORT=8443
export MFLUX_UMASK=0007
export MFLUX_ADMIN=mflux

The startup script modified from $MFLUX_HOME/bin/unix/mediaflux should be saved to /etc/init.d/mediaflux

sudo cp $MFLUX_HOME/bin/unix/mediaflux /etc/init.d/mediaflux

sudo vi /etc/init.d/mediaflux

Note: the following changes is required by systemd on Ubuntu Linux 16.04:

#!/bin/bash

# Required-Start:    $remote_fs $syslog
# Required-Stop:     $remote_fs $syslog
# Default-Start:  2 3 4 5
# Default-Stop: 0 1 6

Note: the following changes is to run the sevice as user mflux rather than root:

start-stop-daemon --start --background --umask $MFLUX_UMASK -c $MFLUX_ADMIN --exec "${JAVA}" -- -jar $MFLUX_HOME/bin/aserver.jar application.home=$MFLUX_HOME nogui $DEBUG >> $MFLUX_HOME/volatile/logs/unix_start.log&

Start Mediaflux server on system reboot using systemd on Ubuntu Linux 16.04

Modify mediaflux script file from $MFLUX_HOME/bin/unix/mediaflux and install it to /etc/init.d/mediaflux.

create systemd unit file at /lib/systemd/system/mediaflux.service

[Unit]
Description=Mediaflux Server
After=remote-fs.target
   
[Service]
Type=forking
ExecStart=/etc/init.d/mediaflux start
ExecStop=/etc/init.d/mediaflux force-stop

[Install]
WantedBy=multi-user.target

Enable the systemd unit (so that it is called on reboot)
```
sudo systemctl enable mediaflux.service
```
Start the service manually(without reboot)
```
sudo systemctl start mediaflux.service
```

Ubuntu Linux 16.04 systemd unit file for Mediaflux server

[Unit]
Description=Mediaflux Server
After=remote-fs.target
   
[Service]
Type=forking
ExecStart=/etc/init.d/mediaflux start
ExecStop=/etc/init.d/mediaflux force-stop

[Install]
# IMPORTANT: The following line is to insert the service to the dependency tree. Otherwise, it will not be called on system reboot.
WantedBy=multi-user.target

maven: include libraries into you local repository

#!/bin/bash
export MFLUX_HOME=/opt/mflux

export MFLUX_VERSION=3.9.011

mvn install:install-file -Dfile=${MFLUX_HOME}/dev/plugin/lib/aplugin.jar -DgroupId=com.arcitecta -DartifactId=aplugin -Dname=aplugin -Dversion=${MFLUX_VERSION} -Dpackaging=jar -DperformRelease=true -DcreateChecksum=true

mvn install:install-file -Dfile=${MFLUX_HOME}/dev/client/java/mfclient.jar -DgroupId=com.arcitecta -DartifactId=mfclient -Dname=mfclient -Dversion=${MFLUX_VERSION} -Dpackaging=jar -DperformRelease=true -DcreateChecksum=true

mvn install:install-file -Dfile=${MFLUX_HOME}/dev/client/gwt/mfclientgwt.jar -DgroupId=com.arcitecta -DartifactId=mfclientgwt -Dname=mfclientgwt -Dversion=${MFLUX_VERSION} -Dpackaging=jar -DperformRelease=true -DcreateChecksum=true

mvn install:install-file -Dfile=${MFLUX_HOME}/dev/client/gwt/mfclientguigwt.jar -DgroupId=com.arcitecta -DartifactId=mfclientguigwt -Dname=mfclientguigwt -Dversion=${MFLUX_VERSION} -Dpackaging=jar -DperformRelease=true -DcreateChecksum=true

After the above step, you can add dependency to your maven project's pom.xml:

    <dependencies>
        <dependency>
            <groupId>com.arcitecta</groupId>
            <artifactId>aplugin</artifactId>
            <version>3.9.011</version>
        </dependency>
    </dependencies>

My Tech Notes

Search This Blog

Mediaflux: service to resolve DNS SRV record

ldap

Kerberos

Enable TLS/SSL encrypted DICOM transfer to Mediaflux server (using DCMTK storescu)

I. TLS/SSL encrypted DICOM transfer to Mediaflux DICOM server without verifying the certificates on both ends

II. TLS/SSL encrypted DICOM transfer to Mediaflux DICOM server with certificate verification on both ends

Mediaflux aterm xvalues: how to handle values contains spaces

see also

Install SSL certificate on Mediaflux server

Mediaflux TCL script: loop through query results with cursors

Mediaflux example: tcl script to repair metadata with empty string as attribute

JavaCC Template to generate Mediaflux asset path

Mediaflux Plugin Development: Create dataset from HTTP URL (with Basic Auth)

Java: HTTP GET data from URL (with basic HTTP Auth)

Multi-threading in Mediaflux Plugin Service

Modify Mediaflux mflux package to install it into non-default schema

Mediaflux startup script to run the server as non-root user

see also

Start Mediaflux server on system reboot using systemd on Ubuntu Linux 16.04

see also

Ubuntu Linux 16.04 systemd unit file for Mediaflux server

see also

maven: include libraries into you local repository

see also