DeveloperTutorial

From EdnaWiki
Revision as of 22:07, 25 September 2010 by Kieffer (talk | contribs) (Run the pipeline)

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

In this tutorial we will learn how to create a custom pipeline for Online Data Analysis (ODA) on a concrete & useful example, even if it is not directly related to synchrotron radiation. This proves EDNA is a generic purpose framework.

At the end of this tutorial you will have learned to

  • Create execution plugins, one running an external program, one written in pure python
  • Re-use execution plugins provided in EDNA toolbox
  • Assemble them in a pipeline by writing a control plugin
  • Run them all on a set of raw-images.


Purpose of the ODA-pipeline

The suggested pipeline is for developing raw images. As the developments operation (conversion raw -> JPEG) can be CPU-consuming (between 1 and 10 seconds per image), using the multi-threading capabilities of EDNA is very interesting. In this pipeline, we will also take care of the metadata associated to the raw-image and transfer them from the raw to the developed image (as Exif tags) and prepare the archiving of the raw file using a compressor like zip or bzip2 (you can win a factor 2 to 3 in the size for archiving).

Online capabilities of EDNA can be checked by running it during a digital camera unloads its images (using gphoto2 for example)

First sub-division of the pipeline

  • Dcraw provides a multi-plateform solution for extracting images from Raw files. It is compatible with most of the models of digital cameras and it is a command line tool, well suited for creating an execution plugin derived from EDPluginExecProcessScript.
  • As DCraw does only the raw development, we will have to re-compress the extracted image in a format suited for web-publication like JPEG. This can be done with a tool like convert from ImageMagick or with the Python Imaging Library. Fortunately, conversion to JPEG are pretty common an such a plugin already exists in the EDNA-toolbox (EDPluginExecThumbnailv10).
  • The transfer of metadata will be made using pyexiv2, a python binding for the C++ library exiv2.
  • Finally we will have to remove some temporary files (directly coded in python) and compress the raw-file, again using an existing plugin called EDPluginExecCommandLinev10.


Before starting

This pipeline is (more or less) available in the SVN of EDNA, this can help you if feel lost in doing this tutorial, but as each plugin is supposed to be unique, we have to start from a snapshot of the repository without the project photov1.

For this download the latest nightly build from ExecPlugins, uncompress it, setup environment variables and run the tests for the kernel and the test for the two plugins from the toolbox we intent to use.

wget http://www.edna-site.org/pub/nightly/EDNA-20100919-ExecPlugins-rev2071.tar.gz
tar -xvzf EDNA-20100919-ExecPlugins-rev2071.tar.gz
export EDNA_HOME=$PWD/edna
export EDNA_SITE=edna-site
alias edt=$EDNA_HOME/kernel/bin/edna-test-launcher.py
mkdir tmp
cd tmp

As their name indicates it, the nightly build changes every night and you should adjust 20100919 to the current day as well as the revision number. If you are behing a firewall you shall setup the proxy by setting the environment variable http_proxy (this is also true under windows !!!)

export http_proxy=http://proxy.institution.org:3128

The name of EDNA_SITE is for the moment not important, it is just a way to tell EDNA what configuration to use. For running EDNA, you will need a python interpreter, between version 2.5 and <3.0. Compatibility with python3 is not yet checked.

To run the tests, launch

edt --test EDTestSuiteKernel
[...]
 [UnitTest]: ###################################################################
 [UnitTest]: Result for EDTestSuiteKernel : SUCCES
 [UnitTest]: 
 [UnitTest]: 
 [UnitTest]:   Total number of test cases executed with SUCCESS : 10
 [UnitTest]:   Total number of test cases executed with FAILURE : 0
 [UnitTest]: 
 [UnitTest]: Total number of test methods executed with SUCCESS : 26
 [UnitTest]: Total number of test methods executed with FAILURE : 0
 [UnitTest]: 
 [UnitTest]:                                            Runtime : 2.537 [s]
 [UnitTest]: ###################################################################
edt --test EDTestSuitePluginExecCommandLinev10
[...]
 [UnitTest]: ###################################################################
 [UnitTest]: Result for EDTestSuitePluginExecCommandLinev10 : SUCCES
 [UnitTest]: 
 [UnitTest]: 
 [UnitTest]:   Total number of test cases executed with SUCCESS : 3
 [UnitTest]:   Total number of test cases executed with FAILURE : 0
 [UnitTest]: 
 [UnitTest]: Total number of test methods executed with SUCCESS : 3
 [UnitTest]: Total number of test methods executed with FAILURE : 0
 [UnitTest]: 
 [UnitTest]:                                            Runtime : 3.857 [s]
 [UnitTest]: ###################################################################
edt --test  EDTestSuitePluginThumbnailv10
[...]
 [UnitTest]: ###################################################################
 [UnitTest]: Result for EDTestSuitePluginThumbnailv10 : SUCCES
 [UnitTest]: 
 [UnitTest]:  Number of executed test suites in this test suite : 1
 [UnitTest]: 
 [UnitTest]: 
 [UnitTest]:   Total number of test cases executed with SUCCESS : 12
 [UnitTest]:   Total number of test cases executed with FAILURE : 0
 [UnitTest]: 
 [UnitTest]: Total number of test methods executed with SUCCESS : 13
 [UnitTest]: Total number of test methods executed with FAILURE : 0
 [UnitTest]: 
 [UnitTest]:                                            Runtime : 10.192 [s]
 [UnitTest]: ###################################################################

If the tests are crashing, please contact the team of EDNA-developers for support !!!

You can look at the log files created and the libraries with all input and output of all plugins. Post-mortem analysis is always possible with EDNA using those logs.

Coding with EDNA

At this point we should mention that EDNA-core-developers are using Eclipse as IDE with PyDev and an UML tool called Enterprise Architect. As the latest is neither running under Unix neither open source, we are working on a replacement UML tool. If you are interested on out coding convention, please refer to our Documentation_for_developers. None of those tools will ever been used during this tutorial (so EDNA still is free !)

Raw development: EDPluginExecDcrawv1_0

This is your first EDNA ExecPlugin, in a sense it is only a wrapper for the dcraw program. At this point you will need to install the program DCRaw.

Under Debian/Ubuntu, simply type:

sudo aptitude install dcraw

Under other operating systems, please refer to the dcraw web site: http://www.cybercom.net/~dcoffin/dcraw/

Datamodel

All input and output of an EDNA plugin is managed by data models which are first represented as UML diagram the exported to XSD and finally converted to python code using meta-programming.

Datamodel definition

The datamodel is a description in graphical form of all inputs and outputs of any plugin. They can be compared with "records" in Pascal or "structures" in C, but they are classes, so they have additional methods (feature) automatically added like import/export in XML, accessors (setters/getters à la java)...

In EDNA we have two kind of class described in this way XSDataInput and XSDataResult, they represent container for data input and data output of any EDNA plugin.

XSDataInputExecDcrawv1 is the name of the input class, it inherits from XSDataInput (or extends it).

Among the attributes of XSDataInputExecDcrawv1, we have the filename of Raw image (rawImagePath) which is the only mandatory attribute. All other attributes (outputPath, extractThumbnail, ...) are optional, identified by their cardinality [0..1] (either 0 or 1 attribute).

XSDataPhotov1.png

Documentation can also be added in the datamodel, here we re-use the man-page of dcraw.

Transcription in XSD

The transcription in XSD is a direct conversion (done inside Enterprise Architect) in XML of the data structure.

<?xml version="1.0" encoding="UTF-8"?>
<xs:schema targetNamespace="http://www.edna-site.org" xmlns:xs="http://www.w3.org/2001/XMLSchema">
	<xs:include schemaLocation="XSDataCommon.xsd"/>
	<xs:element name="XSDataInputExecDcrawv1" type="XSDataInputExecDcrawv1"/>
	<xs:complexType name="XSDataInputExecDcrawv1">
		<xs:complexContent>
			<xs:extension base="XSDataInput">
				<xs:sequence>
					<xs:element name="rawImagePath" type="XSDataFile" minOccurs="1" maxOccurs="1"/>
					<xs:element name="outputPath" type="XSDataFile" minOccurs="0" maxOccurs="1"/>
					<xs:element name="exportTiff" type="XSDataBoolean" minOccurs="0" maxOccurs="1"/>
					<xs:element name="extractThumbnail" type="XSDataBoolean" minOccurs="0" maxOccurs="1"/>
					<xs:element name="whiteBalanceAuto" type="XSDataBoolean" minOccurs="0" maxOccurs="1"/>
					<xs:element name="whiteBalanceFromCamera" type="XSDataBoolean" minOccurs="0" maxOccurs="1"/>
					<xs:element name="levelsFromCamera" type="XSDataBoolean" minOccurs="0" maxOccurs="1"/>
					<xs:element name="interpolationQuality" type="XSDataInteger" minOccurs="0" maxOccurs="1"/>
				</xs:sequence>
			</xs:extension>
		</xs:complexContent>
	</xs:complexType>
	<xs:element name="XSDataResultExecDcrawv1" type="XSDataResultExecDcrawv1"/>
	<xs:complexType name="XSDataResultExecDcrawv1">
		<xs:complexContent>
			<xs:extension base="XSDataResult">
				<xs:sequence>
					<xs:element name="outputPath" type="XSDataFile" minOccurs="1" maxOccurs="1"/>
					<xs:element name="outputFileType" type="XSDataString" minOccurs="0" maxOccurs="1"/>
				</xs:sequence>
			</xs:extension>
		</xs:complexContent>
	</xs:complexType>
 </xs:schema>

Each class (XSDataInputExecDcrawv1 & XSDataResultExecDcrawv1) inherits from a class defined in the datamodel of the kernel (XSDataInput, XSDataResult). You can see that variable have their type defines, like XSDataBoolean, which is also defined the EDNA kernel datamodel (XSDataCommon).

By this way we can:

  • enforce python to define the type of structure exchanged: in python typing is strong but dynamic. Here we manage to have static, strongly typed and expandable types.
  • serialize in XML all input and output of any plugin. This is extremly useful for post-mortem analysis and debugging.

Use the Plugin Generator

The former XSD file is saved in a subdirectory of EDNA:

$EDNA_HOME/photov1/datamodel

For generating the structure of the plugin, EDNA has a plugin generator that we will use. It's documentation can easily be obtained by entering:

$EDNA_HOME/kernel/bin/PluginGenerator.py  --help
Usage: PluginGenerator.py [options]

This program is designed to allow the auto generation of an EDNA plugin.
To use this EDNA_HOME, and EDNA_CONFIG need to be set, where EDNA_CONFIG should be something 
like 'DLS' or 'ESRF' and EDNA_HOME should be set to the direcoty which contains the EDNA 
kernel, it must also contain the base directory structure of the project you want to use.
The default project is EDNA_HOME/templatev1/plugin

Usage :
PluginGenerator.py
	Generates an example plugin in the default project
PluginGenerator.py -n MyPlugin -b Control -v 2.1 -i XSDataInputMTZDUMPUnitCellSpaceGroup -r XSDataResultMTZDUMPUnitCellSpaceGroup -a 'Mark Basham' -c DLS
	Generates a plugin called MyPlugin which is version 2.1 of a control plugin with the specified input and result
	authored by Mark Basham and with DLS as the copyright holder

Options:
 -h, --help            show this help message and exit
 -n PLUGIN_NAME, --plugin-name=PLUGIN_NAME
                       Specifies the name of the plugin you wish to create
 -b BASE_NAME, --base-plugin=BASE_NAME
                       Specifies which plugin you want your new plugin to
                       inherit from
 -p PROJECT_NAME, --project-name=PROJECT_NAME
                       Specifies the name of the project inside edna where this
                       plugin will be i.e. the directory after EDNA_HOME, such
                       as mxv1 or darcv1
 -x XSD_LOCATION, --xsd-location=XSD_LOCATION
                       Specifies the location of the XSD, which is to be used
                       for the plugin. This option is mandatory if the -g tag
                       is not specified (using the local xsd).
 -i XSD_INPUT, --xsd-input=XSD_INPUT
                       Specifies the input XSData class, e.g.
                       XSDataInputStrategy
 -r XSD_RESULT, --xsd-result=XSD_RESULT
                       Specifies the result XSData class, e.g.
                       XSDataResultStrategy
 -g, --xsd-general     Specifies that the plugin generator will be using the
                       general datamodel
 -e PLUGIN_TO_EMULATE, --emulate-plugin=PLUGIN_TO_EMULATE
                       Specifies the location of the plugin which this plugin
                       will replace, this is not curently implemented
 -v PLUGIN_VERSION, --plugin-version=PLUGIN_VERSION
                       Specifies version of the plugin, e.g. 1.0/1.2/2.3/3.10
                       etc
 -a PLUGIN_AUTHOR, --author=PLUGIN_AUTHOR
                       The principle author of the plugin i.e. you
 -c PLUGIN_COPYRIGHT, --copyright=PLUGIN_COPYRIGHT
                       Specifies the copyright string which will be present in
                       the plugin
 -s SLAVE_PLUGIN_NAME, --slave-plugin-name=SLAVE_PLUGIN_NAME
                       If the plugin is a control plugin, this specifies which
                       plugin will be controled
 --site-name=SITE_NAME
                       the name of the site, to decide which configuration file
                       to add the plugin to, i.e. DLS/ESRF etc, default is DLS

So in out example we can type:

$EDNA_HOME/kernel/bin/PluginGenerator.py -b Exec -n Dcraw -v 1.0 -a "Jerome Kieffer" -c "ESRF, Grenoble" -p photov1  --site-name=edna-site -x XSDataPhotov1.xsd -i XSDataInputExecDcrawv1 -r XSDataResultExecDcrawv1

This creates the structure of the plugin:

PluginTree.png

In this tree you can see, in addition to the original XSDataPhotov1.png, .xmi, .EAP and .xsd files, new directories have appeared:

  • conf is populated with XSConfiguration_edna-site.xml. This is the configuration file for your application on your computer. If you chose to install EDNA on a new computer, set your environment variable EDNA_SITE=NewComputer, you will have to create and customize XSConfiguration_NewComputer.xml
  • plugin is populated with a directory called EDPluginDcrawv1_0, containing you plugin and it's tests. Those tests have just been executed and they should be successful.


It's hard to believe but without having coded anything, half of the work is already done, so it's probably time for a coffee-break !

Writing the plugin

Different structure of datamodels in projects

Project usually have many plugins, in our example there will be two execution plugins and one control plugin for chaining them in a pipeline. You could chose to have a common datamodel for all plugins or a separated on, i.e. one datamodel per plugin. The plugin generator assumes a "per-plugin" datamodel, if this option does not please you, here is the way to change it:

Switch to Project datamodel

  • create the directory $EDNA_HOME/photov1/src
  • copy the databinding generator to the project datamodel directory (i.e. from $EDNA_HOME/photov1/plugins/EDPluginExecDcrawv1/datamodel/generateXSDataPhotov1.sh to $EDNA_HOME/photov1/datamodel/generateXSDataPhotov1.sh

this shell script generates the python code equivalent to the datamodel,

  • modify slightly this script, especially those 3 lines, defining the various path.
xsDataBaseName=XSDataPhotov1
xsdHomeDir=${EDNA_HOME}/photov1/datamodel
pyHomeDir=${EDNA_HOME}/photov1/src
  • set this script as executable and run it
chmod +x $EDNA_HOME/photov1/datamodel/generateXSDataPhotov1.sh
cd $EDNA_HOME/photov1/datamodel
./generateXSDataPhotov1.sh

Each time the datamodel (.xsd) is modified, this script should be launched to re-generate the python code corresponding to the datamodel. Now that the XSDataPhotov1 class is in $EDNA_HOME/photov1/src (check), we will modify the existing (but still empty plugin)

First test our (empty-) plugin:

edt --test EDTestSuitePluginExecDcrawv1_0
[...]
 [UnitTest]: Result for EDTestSuitePluginExecDcrawv1_0 : SUCCES
 [UnitTest]: 
 [UnitTest]: 
 [UnitTest]:   Total number of test cases executed with SUCCESS : 2
 [UnitTest]:   Total number of test cases executed with FAILURE : 0
 [UnitTest]: 
 [UnitTest]: Total number of test methods executed with SUCCESS : 2
 [UnitTest]: Total number of test methods executed with FAILURE : 0
 [UnitTest]: 
 [UnitTest]:                                            Runtime : 1.353 [s]
 [UnitTest]: ##################################################################

Delete the datamodel directory with all it contents from the plugin directory and the python class representing it

rm -rf $EDNA_HOME/photov1/plugins/EDPluginExecDcrawv1/datamodel
rm -f $EDNA_HOME/photov1/plugins/EDPluginExecDcrawv1/plugins/XSDataExecDcraw.py*

Replace in the plugin and the tests XSDataExecDcraw by XSDataPhotov1. And now, check that the test still passes:

edt --test EDTestSuitePluginExecDcrawv1_0
[...]
 [UnitTest]: Result for EDTestSuitePluginExecDcrawv1_0 : SUCCES
 [UnitTest]: 
 [UnitTest]: 
 [UnitTest]:   Total number of test cases executed with SUCCESS : 2
 [UnitTest]:   Total number of test cases executed with FAILURE : 0
 [UnitTest]: 
 [UnitTest]: Total number of test methods executed with SUCCESS : 2
 [UnitTest]: Total number of test methods executed with FAILURE : 0
 [UnitTest]: 
 [UnitTest]:                                            Runtime : 1.062 [s]
 [UnitTest]: ###################################################################

Writing tests

In test-driven development you should write the tests then code the functionality that makes the test pass. In EDNA there are two kinds of test cases: unit tests and execution tests as well as test suite which are combination of test suites and/or test cases. We will first define two test cases, write the plugin and finish with the integration of this two test cases in large test suites

Unit tests

In EDNA every plugin is a class that can be instantiated many times. In unit test we usually only check that the class is valid and check the "checkParameter" method of the plugin. This test case is called EDTestCasePluginUnitExecDcrawv1_0.py in the tests/testsuite directory of the plugin. To run the test call:

edt --test EDTestCasePluginUnitExecDcrawv1_0

Execution Tests

This is the point where it starts to be interesting, because we will test that it running ... For running tests we need to prepare data: take a "raw image" and run dcraw on it to develop it.

/usr/bin/dcraw -w -c dcraw.dng > dcraw.ppm

Those two files can be found on http://www.edna-site.org/data/tests/images

Execution test will consist in

* download the raw and the developed image
* run the plugin on the raw image
* compare the result with the developed image downloaded

Edit EDTestCasePluginExecuteExecDcrawv1_0.py, in the class EDTestCasePluginExecuteExecDcrawv1_0, append a method called preProcess that will download the two images:

   def preProcess(self):
       """
       PreProcess of the execution test: download a set of images  from http://www.edna-site.org
       and delete any output file
       """
       EDTestCasePluginExecute.preProcess(self)

       xsDataInput = XSDataInputExecDcrawv1.parseString(self.readAndParseFile(self.getDataInputFile()))
       self.inputFile = xsDataInput.getRawImagePath().getPath().getValue()

       self.xsDataResultReference = XSDataResultExecDcrawv1.parseString(self.readAndParseFile(self.getReferenceDataOutputFile()))
       self.outputFile = self.xsDataResultReference.getOutputPath().getPath().getValue()

       self.loadTestImage([os.path.basename(self.inputFile), os.path.basename(self.outputFile) ])

       if not os.path.isdir(os.path.dirname(self.outputFile)):
           os.makedirs(os.path.dirname(self.outputFile))
       if os.path.isfile(self.outputFile):
           EDVerbose.DEBUG(" Output file exists %s, I will remove it" % self.outputFile)
           os.remove(self.outputFile)

In this piece of code we have used a couple of external modules, we should import them (at the very begining of the file), with the other imports

from XSDataPhotov1 importXSDataInputExecDcrawv1, XSDataResultExecDcrawv1

We also defined 3 instance variable that should be declared in the constructor (__init__):

        self.inputFile = None
        self.outputFile = None
        self.xsDataResultReference=None

Finally we must do a couple of checks, one on the XSDataResult we retrieve from the plugin and one on the output file itself. This is done in testExecute method:

    def testExecute(self):
        """
        """
        self.run()
        plugin = self.getPlugin()
################################################################################
# Compare XSDataResults
################################################################################
        strExpectedOutput = self.readAndParseFile (self.getReferenceDataOutputFile())
        EDVerbose.DEBUG("Checking obtained result...")
        xsDataResultReference = XSDataResultExecDcrawv1.parseString(strExpectedOutput)
        xsDataResultObtained = plugin.getDataOutput()
        EDAssert.strAlmostEqual(xsDataResultReference.marshal(), xsDataResultObtained.marshal(), "XSDataResult output are the same")
################################################################################
# Compare image Files
################################################################################
        outputData = open(xsDataResultObtained.getOutputPath().getPath().getValue(), "rb").read()
        referenceData = open(os.path.join(self.getTestsDataImagesHome(), "dcraw.ppm"), "rb").read()
        EDAssert.strAlmostEqual(referenceData, outputData, _strComment="images are the same")


Nothing of all this will work if we do not provide a correct XML input and output. Skeleton are provided by the plugin generator, we will fill them in:

  • in $EDNA_HOME/photov1/plugins/EDPluginExecDcraw-v1.0/tests/data/XSDataInputDcraw_reference.xml, add:
<?xml version="1.0" ?>
<XSDataInput>
	<rawImagePath>
		<path>
			<value>${EDNA_TESTS_DATA_HOME}/images/dcraw.dng</value>
		</path>
	</rawImagePath>
	<outputPath>
		<path>
			<value>/tmp/edna-${USER}/dcraw.ppm</value>
		</path>
	</outputPath>
</XSDataInput>
  • in $EDNA_HOME/photov1/plugins/EDPluginExecDcraw-v1.0/tests/data/XSDataResult_reference.xml, add:
<?xml version="1.0" ?>
<XSDataResult>
	<outputPath>
		<path>
			<value>/tmp/edna-${USER}/dcraw.ppm</value>
		</path>
	</outputPath>
	<outputFileType>
		<value>ppm</value>
	</outputFileType>
</XSDataResult>

Of course now the tests fail ... because the output is not correct. Indeed nothing runs.

Code the plugin

In this section we modify the plugin itself, located in $EDNA_HOME/photov1/plugins/EDPluginExecDcraw-v1.0/plugins/EDPluginExecDcrawv1_0.py

  • Start by adding all mandatory parameters in the checkParameters method, here only the raw file name.
        self.checkMandatoryParameters(self.getDataInput().getRawImagePath(), "No Raw file provided")
  • Replace everywhere EDPluginExec by EDPluginExecProcessScript because we will call an external program via a script.
  • Remove the process method, we will use the one provided by EDPluginExecProcessScript
  • Define a few variable in the constructor of the class (__init__)
       self.__strRawFile = None
       self.__strOutputType = None
       self.__strOutputFile = None
       self.__bExportTiff = False
       self.__bExtracThumbnail = False
       self.__bWBAuto = False
       self.__bWBCamera = True
       self.__bLevelCamera = False
       self.__iInterpolate = None
  • In the preProcess method, we first read all input data and populate the variable just created; then we call the generateDcrawCommand method:
    def preProcess(self, _edObject=None):
        EDPluginExecProcessScript.preProcess(self)
        EDVerbose.DEBUG("EDPluginExecDcrawv1_0.preProcess")
        self.__strRawFile = self.getDataInput().getRawImagePath().getPath().getValue()
        if self.getDataInput().getOutputPath() is not None:
            self.__strOutputFile = self.getDataInput().getOutputPath().getPath().getValue()
        if self.getDataInput().getExtractThumbnail() is not None:
            self.__bExtracThumbnail = (self.getDataInput().getExtractThumbnail().getValue() in [1, "true", "True", "TRUE", True])
########################################################################
# This option is incompatible with all others
########################################################################
        if self.__bExtracThumbnail is True:
            self.__bWBCamera = False
            self.__strOutputType = None #we cannot know what find of thumbnail is saved, can be jpeg, tiff or nothing !
        else:
            if  self.getDataInput().getExportTiff() is not None:
                self.__bExportTiff = (self.getDataInput().getExportTiff().getValue() in [1, "true", "True", "TRUE", True])
                if self.__bExportTiff:
                    self.__strOutputType = "tiff"
            if self.getDataInput().getWhiteBalanceAuto() is not None:
                self.__bWBAuto = (self.getDataInput().getWhiteBalanceAuto().getValue() in [1, "true", "True", "TRUE", True])
            if self.getDataInput().getWhiteBalanceFromCamera() is not None:
                self.__bWBCamera = (self.getDataInput().getWhiteBalanceFromCamera().getValue() in [1, "true", "True", "TRUE", True])
            if self.getDataInput().getLevelsFromCamera() is not None:
                self.__bLevelCamera = (self.getDataInput().getLevelsFromCamera().getValue()  in [1, "true", "True", "TRUE", True])
            if self.getDataInput().getInterpolationQuality() is not None:
                self.__iInterpolate = self.getDataInput().getInterpolationQuality().getValue()
        self.generateDcrawCommand()

  • Write from scratch the generateCommandLineDcraw method with all options.
    def generateDcrawCommand(self):
        strOptions = "-c" #export to stdout
        if self.__bExtracThumbnail is True:
            strOptions += " -e"
        if self.__bExportTiff is True:
             strOptions += " -T"
        if self.__bWBAuto is True:
            strOptions += " -a"
        if self.__bWBCamera is True:
            strOptions += " -w"
        if self.__bLevelCamera is True:
            strOptions += " -W"
        if self.__iInterpolate is not None:
            strOptions += " -q %s" % self.__iInterpolate
        strOptions += " %s" % self.__strRawFile

        EDVerbose.DEBUG("DCRaw Command LineOption is: %s" % strOptions)
        self.setScriptCommandline(strOptions)


  • As EDNA manages very well the stdout, we will redirect output images to stdout. In the postProcess, we copy the log (catched from stdout) to the output File (if provided)
    def postProcess(self, _edObject=None):
        EDPluginExecProcessScript.postProcess(self)
        EDVerbose.DEBUG("EDPluginExecDcrawv1_0.postProcess")
        # Create some output data
        xsDataResult = XSDataResultExecDcrawv1()
        xsdFile = XSDataFile()
        if self.__strOutputFile is not None:
            import shutil
            shutil.copyfile(os.path.join(self.getWorkingDirectory(),self.getScriptLogFileName()), self.__strOutputFile)
            xsdFile.setPath(XSDataString(self.__strOutputFile))
        else:
            xsdFile.setPath(XSDataString(self.getScriptLogFileName()))
        xsDataResult.setOutputPath(xsdFile)
        if self.__strOutputType is not None:
            xsDataResult.setOutputFileType(XSDataString(self.__strOutputType))
        self.setDataOutput(xsDataResult)
  • In the header section, import the right modules:
import os, shutil
from XSDataCommon import XSDataString, XSDataFile

At this point ... run the tests:

  • Unit test fails because in the test, no rawImageFile is provided (append it)
from XSDataCommon import XSDataFile
[...]
        xsDataInput.setRawInputFile(XSDataFile())

Configure the plugin

As you can see in the log of the test

 [UnitTest]: ###################################################################
 [UnitTest]: Result for EDTestSuitePluginExecDcrawv1_0 : SUCCES
 [UnitTest]: 
 [UnitTest]: 
 [UnitTest]: OBS! The following test cases not executed due to errors:
 [UnitTest]: 
 [UnitTest]:   EDTestCasePluginExecuteExecDcrawv1_0 :
 [UnitTest]:       Missing configuration for EDPluginExecDcrawv1_0
 [UnitTest]: 
 [UnitTest]:            Total number of test cases NOT EXECUTED : 1
 [UnitTest]: 
 [UnitTest]:   Total number of test cases executed with SUCCESS : 1
 [UnitTest]:   Total number of test cases executed with FAILURE : 0
 [UnitTest]: 
 [UnitTest]: Total number of test methods executed with SUCCESS : 1
 [UnitTest]: Total number of test methods executed with FAILURE : 0
 [UnitTest]: 
 [UnitTest]:                                            Runtime : 1.067 [s]
 [UnitTest]: ###################################################################

So lets take the configuration file $EDNA_HOME/photov1/conf/XSConfiguration_edna-site.xml In this file you can configure each plugin with it's executable path, the compatible versions, the time-out (by default 10 mn / 600 sec) ... Here we set only the path to the executable:

<?xml version="1.0" ?>
<XSConfiguration> 
	<XSPluginList>
		<XSPluginItem>
			<name>EDPluginExecDcrawv1_0</name>  
			<XSParamList>
				<XSParamItem>
					<name>execProcessScriptExecutable</name>
					<value>/usr/bin/dcraw</value>
				</XSParamItem>
			</XSParamList>
		</XSPluginItem>
	</XSPluginList> 
</XSConfiguration> 

Now the test are running. if you use the --debug switch in the test command line you get additional information, including profiling info.

Hardening tests

In the execution test we wrote, we compare the output file to a reference file, character by character. A slight difference in those will break the test and this happens all the time when changing CPU architecture, compiler optimization options, ... One solution to go around this is to read the output images as images and then compare images.

For this we will use PIL, the Python Imaging Library, provided either by you operating system, either by EDNA. For this append in the import headers of the execution test:

import sys
from EDUtilsLibraryInstaller    import EDUtilsLibraryInstaller, installLibrary

################################################################################
# AutoBuilder for PIL
################################################################################
architecture = EDUtilsLibraryInstaller.getArchitecture()
imagingPath = os.path.join(os.environ["EDNA_HOME"], "libraries", "20091115-PIL-1.1.7", architecture)

###############################################################################
# Import the right version of PIL
###############################################################################
pydictModulesBeforePIL = sys.modules.copy()
try:
    import Image
except:
    if  os.path.isdir(imagingPath) and (imagingPath not in sys.path):
        sys.path.insert(1,imagingPath)
    else:
        installLibrary(imagingPath)
    import Image

if map(int, Image.VERSION.split(".")) < [1, 1, 6]:
    print "Wrong PIL: Remove old PIL from imported modules"
    for oneModule in sys.modules.copy():
        if oneModule not in pydictModulesBeforePIL:
            del sys.modules[ oneModule ]
    if  os.path.isdir(imagingPath) and (imagingPath not in sys.path):
        sys.path.insert(1, imagingPath)
    else:
        installLibrary(imagingPath)
    import Image

print "Version of Python Imaging Library found: %s" % Image.VERSION
del pydictModulesBeforePIL
import  ImageChops

Then here is the comparison code between two images:

################################################################################
# Compare image Files
################################################################################
        outputData = Image.open(self.outputFile)
        referenceData = Image.open(os.path.join(self.getTestsDataImagesHome(), os.path.basename(self.outputFile)))
        delta = ImageChops.difference(outputData, referenceData)
        deltaColor = delta.getextrema()
        i = 0
        for color in deltaColor:
            EDAssert.lowerThan(color[1], 12, "Maximum tone variation is %s for channel %s" % (color[1], "RGBA"[i]))
            i += 1

We accept that the image made by difference between reference and obtained are limited to 12 unit of color (out of 255) on every channel. This variation is pretty large, largest differences having been observed between computer in 32 and 64 bits.

Test suites

You noticed we did not test all options in dcraw. For this we will create many EDTestCaseExecute with (ideally) all options tested. Then create an EDTestSuiteExecute, chaining all EDTestCaseExecute.

As we do object oriented programming, we can re-use most of the tests and only modify input and output XML strings. Here is the code that extends the former tests using tiff outputs:

import os
from EDTestCasePluginExecuteExecDcrawv1_0        import EDTestCasePluginExecuteExecDcrawv1_0

class EDTestCasePluginExecuteExecDcrawv1_0_tiff(EDTestCasePluginExecuteExecDcrawv1_0):
    """
    Those are all execution tests for the EDNA Exec plugin Dcrawv1_0
    """

    def __init__(self, _strTestName=None):
        """
        """
        EDTestCasePluginExecuteExecDcrawv1_0.__init__(self, "EDPluginExecDcrawv1_0")
        self.setDataInputFile(os.path.join(self.getPluginTestsDataHome(), \
                                           "XSDataInputDcraw_reference_tiff.xml"))
        self.setReferenceDataOutputFile(os.path.join(self.getPluginTestsDataHome(), \
                                                     "XSDataResultDcraw_reference_tiff.xml"))

if __name__ == '__main__':

    testDcrawv1_0instance = EDTestCasePluginExecuteExecDcrawv1_0_tiff("EDTestCasePluginExecuteExecDcrawv1_0_tiff")
    testDcrawv1_0instance.execute()

Transfer metadata: EDPluginCopyExifv1_0

This is your second EDNA plugin, it is very different from the first one as we will write completely in python, relying on an external library called pyexiv2

Under Debian/Ubuntu, simply type:

sudo aptitude install exiv2 python-pyexiv2

Under other operating systems, please install those two libraries from:

Datamodel

Now you are familiar with the datamodels:

UML Diagram

XSDataPhotov1-exif.png

Very simple datamodel: in input two files: the Raw and the Jpeg file, in output only one.

XSD representation

	<xs:element name="XSDataInputCopyExifv1" type="XSDataInputCopyExifv1"/> 
	<xs:complexType name="XSDataInputCopyExifv1">
		<xs:complexContent>
			<xs:extension base="XSDataInput">
				<xs:sequence>
					<xs:element name="inputImagePath" type="XSDataFile" minOccurs="1" maxOccurs="1"/>
					<xs:element name="outputImagePath" type="XSDataFile" minOccurs="1" maxOccurs="1"/>
				</xs:sequence>
			</xs:extension>
		</xs:complexContent>
	</xs:complexType>
	<xs:element name="XSDataResultCopyExifv1" type="XSDataResultCopyExifv1"/>
	<xs:complexType name="XSDataResultCopyExifv1">
		<xs:complexContent>
			<xs:extension base="XSDataResult">
				<xs:sequence>
					<xs:element name="outputImagePath" type="XSDataFile" minOccurs="1" maxOccurs="1"/>
				</xs:sequence>
			</xs:extension>
		</xs:complexContent>
	</xs:complexType>

Once those modification have been integrated in the XSD, regenerate the Python data-binding by re-executing generateXSDataPhotov1.sh

Use the plugin Generator

Go to the datamodel directory and call the plugin generator:

$EDNA_HOME/kernel/bin/PluginGenerator.py  -n CopyExif -v 1.0 -a "Jerome Kieffer" -c "ESRF, Grenoble" -p photov1  --site-name=edna-site -x XSDataPhotov1.xsd -i XSDataInputCopyExifv1 -r XSDataResultCopyExifv1

Then clean up a bit:

  • remove the datamodel directory of the plugin, we are using the datamodel at the project level
  • rename the plugin from ExecCopyExif to CopyExif (it does not execute anything). There are 4 files to modify. Do the same with the tests
  • replace all import from XSDataCopyExifv1 by XSDataPhotov1.py
  • Remove the $EDNA_HOME/.XSDataDictionaryModule.xml that keeps track of all plugins.


Code the plugin

  • In the header, import the two libraries pyexiv2 and os
  • In the Constructor, define 2 instance attributes for the names and two for the exifs:
        self.strInfile = None
        self.strOutfile = None
        self.exifInfile = None
        self.exifOutpfile = None

  • In the checkParameters method, append:
        self.checkMandatoryParameters(self.getDataInput().getInputImagePath(),"No Input Image Provided")
        self.checkMandatoryParameters(self.getDataInput().getOutputImagePath(),"No Output Image Provided")
  • In the preProcess method, read the name of the two files and check that they actually exists:
        self.strInfile = self.getDataInput().getInputImagePath().getPath().getValue()
        self.strOutfile = self.getDataInput().getOutputImagePath().getPath().getValue()
        for oneFile in [self.strInfile,self.strOutfile]:
            if not os.path.isfile(oneFile):
                strErrorMessage = EDMessage.ERROR_CANNOT_READ_FILE_02 % (self.getPluginName() + ".preProcess", oneFile)
                EDVerbose.error(strErrorMessage)
                self.addErrorMessage(strErrorMessage)
                raise RuntimeError, strErrorMessage
  • In the process method, open both exifs structures, copy the one from input to output. Save output.
        self.exifInfile= pyexiv2.Image(self.strInfile)
        self.exifOutfile = pyexiv2.Image(self.strOutfile)
        self.exifInfile.readMetadata()
        self.exifOutfile.readMetadata()
        #dcraw sets automatically the good orientation
        self.exifOutfile['Exif.Image.Orientation'] = 1 
        # save the name of the raw image
        self.exifOutfile["Exif.Photo.UserComment"] = self.strInfile 
        for metadata in [ 'Exif.Image.Make', 'Exif.Image.Model', 'Exif.Photo.DateTimeOriginal', 'Exif.Photo.ExposureTime', 'Exif.Photo.FNumber', 'Exif.Photo.ExposureBiasValue', 'Exif.Photo.Flash', 'Exif.Photo.FocalLength', 'Exif.Photo.ISOSpeedRatings']:
            try:
                self.exifOutfile[metadata] = self.exifInfile[metadata]
            except:
                EDVerbose.Warning("Error in copying metadata %s in file %s, value: %s" % (metadata, self.strInfile, self.exifInfile[metadata]))
        self.exifOutfile.writeMetadata()
  • In the postProcess, just report the name if the output file.

write the tests

Whole pipeline: EDPluginDevelopRawv1_0

Datamodel

As for execution plugins, control plugins (pipelines) have datamodels, both for input and for outputs.

UML diagram

XSDataPhotov1-Devel.png

In this datamodel, the only mandatory part is the input Raw file. There are a few optional parameters to specify the ouput filename or if you want to modify slightly the behaviour of the plugin.

XSD structure representation

	<xs:element name="XSDataInputDevelopRawv1" type="XSDataInputDevelopRawv1"/>
	<xs:complexType name="XSDataInputDevelopRawv1">
		<xs:complexContent>
 			<xs:extension base="XSDataInput">
				<xs:sequence>
					<xs:element name="inputRaw" type="XSDataFile" minOccurs="1" maxOccurs="1"/>
					<xs:element name="outputJpeg" type="XSDataFile" minOccurs="0" maxOccurs="1"/>
					<xs:element name="copyExifTag" type="XSDataBoolean" minOccurs="0" maxOccurs="1"/>
					<xs:element name="compressRaw" type="XSDataBoolean" minOccurs="0" maxOccurs="1"/>
					<xs:element name="cleanUp" type="XSDataBoolean" minOccurs="0" maxOccurs="1"/>
				</xs:sequence>
			</xs:extension>
		</xs:complexContent>
	</xs:complexType>
	<xs:element name="XSDataResultDevelopRawv1" type="XSDataResultDevelopRawv1"/>
	<xs:complexType name="XSDataResultDevelopRawv1">
		<xs:complexContent>
			<xs:extension base="XSDataResult">
				<xs:sequence>
					<xs:element name="outputFile" type="XSDataFile" minOccurs="1" maxOccurs="1"/>
				</xs:sequence>
			</xs:extension>
		</xs:complexContent>
	</xs:complexType>

Once those modification have been integrated in the XSD, regenerate the Python data-binding by re-executing generateXSDataPhotov1.sh

Run the pipeline

An automatic launcher can be built in inheriting from EDParallelExecute. The only thing to do is provide a function, that converts an input filename into an acceptable XML string for the plugin.

def fileName2xml(filename):
    """Here we create the XML string to be passed to the EDNA plugin from the input filename
    This can / should be modified by the final user
    
    @param filename: full path of the input file
    @type filename: python string representing the path
    @rtype: XML string
    @return: python string  
    """
    basename,extention = os.path.splitext(filename)
    if extension.lower() in [".cr2", ".arw", ".mrw", ".dng"]:
        xml = "<XSDataInput>\
    <inputRaw>\
        <path>\
            <value>%s</value>\
        </path>\
    </inputRaw> \
    <outputJpeg>\
        <path>\
            <value>%s-raw.jpg</value>\
        </path>\
    </outputJpeg>\
</XSDataInput>" % (filename,basename)
    else:
        xml = "<XSDataInput></XSDataInput>"
    return xml

Run offline the pipeline

Run online the pipeline

Use the command line to launch the pipeline

Create a Device Server Tango to launch the pipeline