Skip to content

IDFLoadOrder.md

Latest commit

 

History

History
182 lines (129 loc) · 9.84 KB

IDFLoadOrder.md

File metadata and controls

182 lines (129 loc) · 9.84 KB
Raw

Motivation

Nexus files have three pieces of information in which we can use to determine which IDF to load. The current structure is poorly understood, and on investigation is a mess.

At the moment, for determining the mangled filename (how we decide if the file is already loaded in the IDS):

  1. if the filename is present use the file contents if it is available
  2. use the xml string for loading the file from nexus

and for loading the file

  1. if instrument name is not set, undefined behaviour
  2. if instrument_xml is present then load that xml (instrument filename is set to the value in the NXS file, not the nxs file itself)
  3. if filename has been entered them make up a filename out of the directory and instrument name and output an incorrect log message. Load the most fitting file based on the instrument name

At no point would the filename specified be loaded even if correct and present

Proposed Solution

For file name mangling:

  1. use the xml string for loading the file from nexus
  2. if the filename is present use the file contents if it is available

for loading the file:

  1. if instrument_xml is present then load that xml (instrument filename is set to the nxs file itself)
  2. if filename has been entered them Load the file from that location is it is present.
    1. on the same path
    2. in any of the instrument directories (in the correct order)
  3. for both of the above if the instrument name is not set then read it from the inst defnintion.
  4. Load the most fitting file based on the instrument name

This primarily involves changes to:

  • ExperimentInfo::loadInstrumentInfoNexus - for the loading order of the IDF itself
  • InstrumentDefinitionParser::getMangledName - for the order of determining the mangled name

Additional Comments

* The rules for getting the instrument xml for the checksum and when performing the actual load must be identical.
* The checksum is generated by ChecksumHelper and is a SHA-1 of the file contents (after conversion to linux file endings, and trimming at both ends).
* We can always use LoadInstrument to overwrite the IDF contained in the nexus file if we have something more up-to-date in an xml file. 
* I think the instrument information in the MantidPlot Workspace List should indicate exactly where the IDF information came from, especially if it came from the nexus file itself.
* We should deprecate and remove (as part of this work) any flags or fields that are not used, or at the very least, remove the code that reads them in mantid with comments explaining why this isn’t used. Deprecation is fine, although removing unused flags from IDF files cannot be done within the same release (as the IDF’s go out to users immediately), I have raised a ticket for the next release.
* We need to capture good user documentation to explain the logic for the IDF reading. I quite often get instrument scientists asking me this.

Email tread for this core change suggestion

Email tread, which includes an agreed conclusion, for this core change suggesion.

Hi,

For the record.  

Following the emails below, Nick, Owen and myself discuss this further, and it was agreed to take out step 2 and 3, so that the order of loading for IDFs from nexus files is:

1.	Xml source in the nexus file if present
2.	Otherwise IDF xml file in the instrument directories (including instrument repository) based on the stored instrument name

Anders

From: mantid-tech-bounces@mantidproject.org [mailto:mantid-tech-bounces@mantidproject.org] On Behalf Of nick.draper@stfc.ac.uk
Sent: 21 May 2015 14:15
To: Arnold, Owen (Tessella,RAL,ISIS); mantid-tech@mantidproject.org
Subject: Re: [Mantid-tech] Core change suggestion for the order of Loading for IDFs from nexus files

Owen,

For the name mangling:
The mangled mane is only used as the key for storage and lookups in the IDS.  
I have put a change in to make that use InstName+checksum rather than InstName+LastModifiedDate, but that is a separate ticket and PR to this suggestion. While doing that work I saw the larger mess that this is intended to address.

You summary is very close but misses a final point.

1.	Xml source in the nexus file if present
2.	Otherwise IDF xml file in the local directory based on the instrument/name in nexus file
3.	Otherwise IDF xml file in the instrument directories (including instrument repository) based on the stored file name
4.	Otherwise IDF xml file in the instrument directories (including instrument repository) based on the stored instrument

For your other points I agree,
•	The rules for getting the instrument xml for the checksum and when performing the actual load must be identical. 
The checksum is generated by ChecksumHelper and is a SHA-1 of the file contents (after conversion to linux file endings, and trimming at both ends).
•	We can always use LoadInstrument to overwrite the IDF contained in the nexus file if we have something more up-to-date in an xml file.
I agree, I will test the LoadInstrument case
•	I think the instrument information in the MantidPlot Workspace List should indicate exactly where the IDF information came from, especially if it came from the nexus file itself
I agree, although that is wrong for nexus files at present
•	We should deprecate and remove (as part of this work) any flags or fields that are not used, or at the very least, remove the code that reads them in mantid with comments explaining why this isn’t used
Deprecation is fine, although removing unused flags from IDF files cannot be done within the same release (as the IDF’s go out to users immediately), I have raised a ticket for the next release.
•	We need to capture good user documentation to explain the logic for the IDF reading. I quite often get instrument scientists asking me this.
I agree, this should be a requirement of this work.

Regards,
Nick Draper

From: Arnold, Owen (Tessella,RAL,ISIS) 
Sent: 21 May 2015 13:26
To: Draper, Nick (-,RAL,ISIS); mantid-tech@mantidproject.org
Subject: RE: Core change suggestion for the order of Loading for IDFs from nexus files

I agree, based on your description that we have a problem here.

Is it just the name that gets mangled? I thought that a checksum would be generated from the entire idf source. The following assumes the latter.

Just to be clear, you are proposing for both the checksum generation and the idf loading that the order of precedence and that order is:

1.	Xml source in the nexus file if present
2.	Otherwise IDF xml file in the local directory based on the instrument/name in nexus file
3.	Otherwise IDF xml file in the instrument directories (including instrument repository) 

If so, I think that would be a sensible order. Some additional things are:

•	The rules for getting the instrument xml for the checksum and when performing the actual load must be identical. 
•	We can always use LoadInstrument to overwrite the IDF contained in the nexus file if we have something more up-to-date in an xml file.
•	I think the instrument information in the MantidPlot Workspace List should indicate exactly where the IDF information came from, especially if it came from the nexus file itself
•	We should deprecate and remove (as part of this work) any flags or fields that are not used, or at the very least, remove the code that reads them in mantid with comments explaining why this isn’t used
•	We need to capture good user documentation to explain the logic for the IDF reading. I quite often get instrument scientists asking me this.

Owen.

From: mantid-developers-bounces@mantidproject.org [mailto:mantid-developers-bounces@mantidproject.org] On Behalf Of nick.draper@stfc.ac.uk
Sent: 21 May 2015 11:39
To: Mantid Developers
Subject: [Mantid-developers] FW: Core change suggestion for the order of Loading for IDFs from nexus files

Just to be clear, this is a proposed change that requires attention and approval from the TSC.

Regards,
Nick Draper

From: Draper, Nick (-,RAL,ISIS) 
Sent: 20 May 2015 16:02
To: Mantid Developers
Subject: Core change suggestion for the order of Loading for IDFs from nexus files

http://trac.mantidproject.org/mantid/ticket/11818

Reason:
Nexus files have three pieces of information in which we can use to determine which IDF to load.
The current structure is poorly understood, and on investigation is a mess.


At the moment:
For determining the mangled filename (how we decide if the file is already loaded in the IDS)
1.	If the filename is present use the file contents if it is available
2.	use the xml string for loading the file from nexus
for loading the file
1.	if instrument name is not set, undefined behaviour
2.	if instrument_xml is present then load that xml (instrument filename is set to the value in the NXS file, not the nxs file itself) 
1.	if filename has been entered them make up a filename out of the directory and instrument name and output an incorrect log message.
3.	Load the most fitting file based on the instrument name
•	At no point would the filename specified be loaded even if correct and present

I suggest we change this to:
For file name mangling:
1.	use the xml string for loading the file from nexus
2.	If the filename is present use the file contents if it is available
for loading the file:
1.	if instrument_xml is present then load that xml (instrument filename is set to the nxs file itself)
2.	if filename has been entered them Load the file from that location is it is present. 
1.	on the same path
2.	in any of the instrument directories (in the correct order)
3.	for both of the above if the instrument name is not set then read it from the inst defnintion.
4.	Load the most fitting file based on the instrument name


This primarily involves changes to:
•	ExperimentInfo::loadInstrumentInfoNexus - for the loading order of the IDF itself
•	InstrumentDefinitionParser::getMangledName - for the order of determining the mangled name