How to build parallel HDF5 with Intel MPI, Intel C++ and Intel Fortran

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

How to build parallel HDF5 with Intel MPI, Intel C++ and Intel Fortran

Tony Garratt
Following on my previous post, I notice that at


you have built with Intel Fortran. How would one use cmake to build parallel HDF5 on Windows with both Intel C++ and Intel Fortran please?

Regards,
Tony

--
Dr Tony Garratt
Tel: +44 7624 309933

_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How to build parallel HDF5 with Intel MPI, Intel C++ and Intel Fortran

Samer Afach-4
Hi Tony:

I don't know what the features of hdf5 with parallel are, but if all you want to achieve is have thread safety, I recommend that you use mutexes manually. Just wrap every call to hdf5 with std::lock_guard() or std::unique_lock(). If you can't use C++11, then you can get the same features from the boost library. This will make your code work independent of the build settings of hdf5.

I do this to avoid the obvious mistake of running a program in parallel with a version of hdf5 that is built for serial. It's much cleaner.

Cheers,
Sam

On February 21, 2017 12:23:03 PM GMT+01:00, Tony Garratt <[hidden email]> wrote:
Following on my previous post, I notice that at


you have built with Intel Fortran. How would one use cmake to build parallel HDF5 on Windows with both Intel C++ and Intel Fortran please?

Regards,
Tony

--
Dr Tony Garratt
Tel: +44 7624 309933

--
Sent from my Android device.
_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How to build parallel HDF5 with Intel MPI, Intel C++ and Intel Fortran

Tony Garratt

Thank you for your reply. What I am wanting to do is build CGNS (https://cgns.github.io/WhatIsCGNS.html) in parallel to get parallel I/O, which requires HDF5 to be built in parallel - its not just thread safety I am after. 

Everything works fine on Linux, since there I can still use configure to build HDF5 and even Intel give advice on how to build it with their compilers at https://software.intel.com/en-us/articles/performance-tools-for-software-developers-building-hdf5-with-intel-compilers.

The problem is that the move to cmake has left me in the dark now. If configure was still supported on Windows for HDF5, then I the path forward is known. cmake appears to be very VS centric, which is fine if you are OK with MS compilers and not clear to me how I can switch to Intel compilers.

Regards,
Tony



On Tue, Feb 21, 2017 at 11:58 AM, Samer Afach <[hidden email]> wrote:
Hi Tony:

I don't know what the features of hdf5 with parallel are, but if all you want to achieve is have thread safety, I recommend that you use mutexes manually. Just wrap every call to hdf5 with std::lock_guard() or std::unique_lock(). If you can't use C++11, then you can get the same features from the boost library. This will make your code work independent of the build settings of hdf5.

I do this to avoid the obvious mistake of running a program in parallel with a version of hdf5 that is built for serial. It's much cleaner.

Cheers,
Sam

On February 21, 2017 12:23:03 PM GMT+01:00, Tony Garratt <[hidden email]> wrote:
Following on my previous post, I notice that at


you have built with Intel Fortran. How would one use cmake to build parallel HDF5 on Windows with both Intel C++ and Intel Fortran please?

Regards,
Tony

--
Dr Tony Garratt
Tel: <a href="tel:+44%2076%202430%209933" value="+447624309933" target="_blank">+44 7624 309933

--
Sent from my Android device.

_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5



--
Dr Tony Garratt
Tel: +44 7624 309933

_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [FORGED] Re: How to build parallel HDF5 with Intel MPI, Intel C++ and Intel Fortran

David Brooks
Hi Tony,

You might have already explored this approach but have you seen http://stackoverflow.com/questions/27623110/how-to-generate-a-visual-studio-project-that-uses-the-intel-compiler-using-cmake ??


Regards,
Dave

On 22/02/17 1:06 AM, Tony Garratt wrote:

Thank you for your reply. What I am wanting to do is build CGNS (https://cgns.github.io/WhatIsCGNS.html) in parallel to get parallel I/O, which requires HDF5 to be built in parallel - its not just thread safety I am after. 

Everything works fine on Linux, since there I can still use configure to build HDF5 and even Intel give advice on how to build it with their compilers at https://software.intel.com/en-us/articles/performance-tools-for-software-developers-building-hdf5-with-intel-compilers.

The problem is that the move to cmake has left me in the dark now. If configure was still supported on Windows for HDF5, then I the path forward is known. cmake appears to be very VS centric, which is fine if you are OK with MS compilers and not clear to me how I can switch to Intel compilers.

Regards,
Tony



On Tue, Feb 21, 2017 at 11:58 AM, Samer Afach <[hidden email]> wrote:
Hi Tony:

I don't know what the features of hdf5 with parallel are, but if all you want to achieve is have thread safety, I recommend that you use mutexes manually. Just wrap every call to hdf5 with std::lock_guard() or std::unique_lock(). If you can't use C++11, then you can get the same features from the boost library. This will make your code work independent of the build settings of hdf5.

I do this to avoid the obvious mistake of running a program in parallel with a version of hdf5 that is built for serial. It's much cleaner.

Cheers,
Sam

On February 21, 2017 12:23:03 PM GMT+01:00, Tony Garratt <[hidden email]> wrote:
Following on my previous post, I notice that at


you have built with Intel Fortran. How would one use cmake to build parallel HDF5 on Windows with both Intel C++ and Intel Fortran please?

Regards,
Tony

--
Dr Tony Garratt
Tel: <a moz-do-not-send="true" href="tel:+44%2076%202430%209933" value="+447624309933" target="_blank">+44 7624 309933

--
Sent from my Android device.

_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5



--
Dr Tony Garratt
Tel: +44 7624 309933


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How to build parallel HDF5 with Intel MPI, Intel C++ and Intel Fortran

Matthieu Brucher
In reply to this post by Tony Garratt
As far as I know, you can't have parallel HDF5 on Windows.

Cheers,

Matthieu

2017-02-21 12:23 GMT+01:00 Tony Garratt <[hidden email]>:
Following on my previous post, I notice that at


you have built with Intel Fortran. How would one use cmake to build parallel HDF5 on Windows with both Intel C++ and Intel Fortran please?

Regards,
Tony

--
Dr Tony Garratt
Tel: <a href="tel:+44%2076%202430%209933" value="+447624309933" target="_blank">+44 7624 309933

_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5



--
Information System Engineer, Ph.D.
Blog: http://blog.audio-tk.com/
LinkedIn: http://www.linkedin.com/in/matthieubrucher

_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [FORGED] Re: How to build parallel HDF5 with Intel MPI, Intel C++ and Intel Fortran

Tony Garratt
In reply to this post by David Brooks
Hi Dave
 No - I had not seen that web page, even though I have been doing some web searches. Thank you very much!
Regards,
Tony

On Tue, Feb 21, 2017 at 7:35 PM, David Brooks <[hidden email]> wrote:
Hi Tony,

You might have already explored this approach but have you seen http://stackoverflow.com/questions/27623110/how-to-generate-a-visual-studio-project-that-uses-the-intel-compiler-using-cmake ??


Regards,
Dave

On 22/02/17 1:06 AM, Tony Garratt wrote:

Thank you for your reply. What I am wanting to do is build CGNS (https://cgns.github.io/WhatIsCGNS.html) in parallel to get parallel I/O, which requires HDF5 to be built in parallel - its not just thread safety I am after. 

Everything works fine on Linux, since there I can still use configure to build HDF5 and even Intel give advice on how to build it with their compilers at https://software.intel.com/en-us/articles/performance-tools-for-software-developers-building-hdf5-with-intel-compilers.

The problem is that the move to cmake has left me in the dark now. If configure was still supported on Windows for HDF5, then I the path forward is known. cmake appears to be very VS centric, which is fine if you are OK with MS compilers and not clear to me how I can switch to Intel compilers.

Regards,
Tony



On Tue, Feb 21, 2017 at 11:58 AM, Samer Afach <[hidden email]> wrote:
Hi Tony:

I don't know what the features of hdf5 with parallel are, but if all you want to achieve is have thread safety, I recommend that you use mutexes manually. Just wrap every call to hdf5 with std::lock_guard() or std::unique_lock(). If you can't use C++11, then you can get the same features from the boost library. This will make your code work independent of the build settings of hdf5.

I do this to avoid the obvious mistake of running a program in parallel with a version of hdf5 that is built for serial. It's much cleaner.

Cheers,
Sam

On February 21, 2017 12:23:03 PM GMT+01:00, Tony Garratt <[hidden email]> wrote:
Following on my previous post, I notice that at


you have built with Intel Fortran. How would one use cmake to build parallel HDF5 on Windows with both Intel C++ and Intel Fortran please?

Regards,
Tony

--
Dr Tony Garratt
Tel: <a href="tel:+44%2076%202430%209933" value="+447624309933" target="_blank">+44 7624 309933

--
Sent from my Android device.

_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5



--
Dr Tony Garratt
Tel: +44 7624 309933


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5



--
Dr Tony Garratt
Tel: +44 7624 309933

_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [FORGED] Re: How to build parallel HDF5 with Intel MPI, Intel C++ and Intel Fortran

Tony Garratt
In reply to this post by David Brooks

Thank you. This link also refers to using the -T option that I have used (with some partial success with Intel XE2015). I tried the suggestion on the page but it did not work for me. However, I think the way forward is to use -T option and then manually open the overall VS project and build manually.

Regards,
Tony

On Tue, Feb 21, 2017 at 7:35 PM, David Brooks <[hidden email]> wrote:
Hi Tony,

You might have already explored this approach but have you seen http://stackoverflow.com/questions/27623110/how-to-generate-a-visual-studio-project-that-uses-the-intel-compiler-using-cmake ??


Regards,
Dave

On 22/02/17 1:06 AM, Tony Garratt wrote:

Thank you for your reply. What I am wanting to do is build CGNS (https://cgns.github.io/WhatIsCGNS.html) in parallel to get parallel I/O, which requires HDF5 to be built in parallel - its not just thread safety I am after. 

Everything works fine on Linux, since there I can still use configure to build HDF5 and even Intel give advice on how to build it with their compilers at https://software.intel.com/en-us/articles/performance-tools-for-software-developers-building-hdf5-with-intel-compilers.

The problem is that the move to cmake has left me in the dark now. If configure was still supported on Windows for HDF5, then I the path forward is known. cmake appears to be very VS centric, which is fine if you are OK with MS compilers and not clear to me how I can switch to Intel compilers.

Regards,
Tony



On Tue, Feb 21, 2017 at 11:58 AM, Samer Afach <[hidden email]> wrote:
Hi Tony:

I don't know what the features of hdf5 with parallel are, but if all you want to achieve is have thread safety, I recommend that you use mutexes manually. Just wrap every call to hdf5 with std::lock_guard() or std::unique_lock(). If you can't use C++11, then you can get the same features from the boost library. This will make your code work independent of the build settings of hdf5.

I do this to avoid the obvious mistake of running a program in parallel with a version of hdf5 that is built for serial. It's much cleaner.

Cheers,
Sam

On February 21, 2017 12:23:03 PM GMT+01:00, Tony Garratt <[hidden email]> wrote:
Following on my previous post, I notice that at


you have built with Intel Fortran. How would one use cmake to build parallel HDF5 on Windows with both Intel C++ and Intel Fortran please?

Regards,
Tony

--
Dr Tony Garratt
Tel: <a href="tel:+44%2076%202430%209933" value="+447624309933" target="_blank">+44 7624 309933

--
Sent from my Android device.

_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5



--
Dr Tony Garratt
Tel: +44 7624 309933


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5



--
Dr Tony Garratt
Tel: +44 7624 309933

_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [FORGED] Re: How to build parallel HDF5 with Intel MPI, Intel C++ and Intel Fortran

Matt Peterson
Hi Tony,

Were you able to get this set up?

The HDF Cmake file continues to be a mystery to me and I've no time to figure it out right now. As an alternative, I've compiled the serial version on Windows using only the command line shell, without Visual Studio. However, if the parallel version compiled for you, I might give it another go.

Thanks,
MP

On Sun, Feb 26, 2017 at 2:11 PM, Tony Garratt <[hidden email]> wrote:

Thank you. This link also refers to using the -T option that I have used (with some partial success with Intel XE2015). I tried the suggestion on the page but it did not work for me. However, I think the way forward is to use -T option and then manually open the overall VS project and build manually.

Regards,
Tony

On Tue, Feb 21, 2017 at 7:35 PM, David Brooks <[hidden email]> wrote:
Hi Tony,

You might have already explored this approach but have you seen http://stackoverflow.com/questions/27623110/how-to-generate-a-visual-studio-project-that-uses-the-intel-compiler-using-cmake ??


Regards,
Dave

On 22/02/17 1:06 AM, Tony Garratt wrote:

Thank you for your reply. What I am wanting to do is build CGNS (https://cgns.github.io/WhatIsCGNS.html) in parallel to get parallel I/O, which requires HDF5 to be built in parallel - its not just thread safety I am after. 

Everything works fine on Linux, since there I can still use configure to build HDF5 and even Intel give advice on how to build it with their compilers at https://software.intel.com/en-us/articles/performance-tools-for-software-developers-building-hdf5-with-intel-compilers.

The problem is that the move to cmake has left me in the dark now. If configure was still supported on Windows for HDF5, then I the path forward is known. cmake appears to be very VS centric, which is fine if you are OK with MS compilers and not clear to me how I can switch to Intel compilers.

Regards,
Tony



On Tue, Feb 21, 2017 at 11:58 AM, Samer Afach <[hidden email]> wrote:
Hi Tony:

I don't know what the features of hdf5 with parallel are, but if all you want to achieve is have thread safety, I recommend that you use mutexes manually. Just wrap every call to hdf5 with std::lock_guard() or std::unique_lock(). If you can't use C++11, then you can get the same features from the boost library. This will make your code work independent of the build settings of hdf5.

I do this to avoid the obvious mistake of running a program in parallel with a version of hdf5 that is built for serial. It's much cleaner.

Cheers,
Sam

On February 21, 2017 12:23:03 PM GMT+01:00, Tony Garratt <[hidden email]> wrote:
Following on my previous post, I notice that at


you have built with Intel Fortran. How would one use cmake to build parallel HDF5 on Windows with both Intel C++ and Intel Fortran please?

Regards,
Tony

--
Dr Tony Garratt
Tel: <a href="tel:+44%2076%202430%209933" value="+447624309933" target="_blank">+44 7624 309933

--
Sent from my Android device.

_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5



--
Dr Tony Garratt
Tel: +44 7624 309933


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5



--
Dr Tony Garratt
Tel: <a href="tel:+44%2076%202430%209933" value="+447624309933" target="_blank">+44 7624 309933

_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [FORGED] Re: How to build parallel HDF5 with Intel MPI, Intel C++ and Intel Fortran

Tony Garratt
​​Hi Matt

 Yes - I did manage to get it to work with Intel compilers on Windows - both serial and parallel builds. What compilers are you using?  Instructions for the Intel compilers are below.



Attached is a new CTestScript.cmake file.  Replace the one you are currently using with this one.

Also, you will need to edit the HDF5config.cmake file and add this
line to it. Change its value to match your Intel compiler:

set(CMAKE_GENERATOR_TOOLSET "Intel C++ Compiler 17.0")

(the name changes depending on what version of Intel compiler you are using - e.g. for XE2015 it is ""Intel C++ Compiler XE 15.0"). 

Also in the config file to build parallel add the following to the config file (this is on the HDF5 web page as well)

set(ADD_BUILD_OPTIONS "${ADD_BUILD_OPTIONS} -DHDF5_ENABLE_PARALLEL:BOOL=ON")
set(ADD_BUILD_OPTIONS "${ADD_BUILD_OPTIONS} -DMPIEXEC_MAX_NUMPROCS:STRING=4")

With Intel compilers I could not build the C++ library, so I added 

  set(ADD_BUILD_OPTIONS "${ADD_BUILD_OPTIONS} -DHDF5_BUILD_CPP_LIB:BOOL=OFF")

after the line

### uncomment/comment and change the following lines for other configuration options

in the config file.

I hope this helps. I have not built the parallel version with VS, only Intel, but I believe you should be OK with VS as long as you have MS MPI.

Regards,
Tony


On Wed, Mar 22, 2017 at 6:25 PM, Matt Peterson <[hidden email]> wrote:
Hi Tony,

Were you able to get this set up?

The HDF Cmake file continues to be a mystery to me and I've no time to figure it out right now. As an alternative, I've compiled the serial version on Windows using only the command line shell, without Visual Studio. However, if the parallel version compiled for you, I might give it another go.

Thanks,
MP

On Sun, Feb 26, 2017 at 2:11 PM, Tony Garratt <[hidden email]> wrote:

Thank you. This link also refers to using the -T option that I have used (with some partial success with Intel XE2015). I tried the suggestion on the page but it did not work for me. However, I think the way forward is to use -T option and then manually open the overall VS project and build manually.

Regards,
Tony

On Tue, Feb 21, 2017 at 7:35 PM, David Brooks <[hidden email]> wrote:
Hi Tony,

You might have already explored this approach but have you seen http://stackoverflow.com/questions/27623110/how-to-generate-a-visual-studio-project-that-uses-the-intel-compiler-using-cmake ??


Regards,
Dave

On 22/02/17 1:06 AM, Tony Garratt wrote:

Thank you for your reply. What I am wanting to do is build CGNS (https://cgns.github.io/WhatIsCGNS.html) in parallel to get parallel I/O, which requires HDF5 to be built in parallel - its not just thread safety I am after. 

Everything works fine on Linux, since there I can still use configure to build HDF5 and even Intel give advice on how to build it with their compilers at https://software.intel.com/en-us/articles/performance-tools-for-software-developers-building-hdf5-with-intel-compilers.

The problem is that the move to cmake has left me in the dark now. If configure was still supported on Windows for HDF5, then I the path forward is known. cmake appears to be very VS centric, which is fine if you are OK with MS compilers and not clear to me how I can switch to Intel compilers.

Regards,
Tony



On Tue, Feb 21, 2017 at 11:58 AM, Samer Afach <[hidden email]> wrote:
Hi Tony:

I don't know what the features of hdf5 with parallel are, but if all you want to achieve is have thread safety, I recommend that you use mutexes manually. Just wrap every call to hdf5 with std::lock_guard() or std::unique_lock(). If you can't use C++11, then you can get the same features from the boost library. This will make your code work independent of the build settings of hdf5.

I do this to avoid the obvious mistake of running a program in parallel with a version of hdf5 that is built for serial. It's much cleaner.

Cheers,
Sam

On February 21, 2017 12:23:03 PM GMT+01:00, Tony Garratt <[hidden email]> wrote:
Following on my previous post, I notice that at


you have built with Intel Fortran. How would one use cmake to build parallel HDF5 on Windows with both Intel C++ and Intel Fortran please?

Regards,
Tony

--
Dr Tony Garratt
Tel: <a href="tel:+44%2076%202430%209933" value="+447624309933" target="_blank">+44 7624 309933

--
Sent from my Android device.

_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5



--
Dr Tony Garratt
Tel: +44 7624 309933


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5



--
Dr Tony Garratt
Tel: <a href="tel:+44%2076%202430%209933" value="+447624309933" target="_blank">+44 7624 309933

_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5



--
Dr Tony Garratt
Tel: <a href="tel:+44%2076%202430%209933" value="+447624309933" target="_blank">+44 7624 309933

_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [FORGED] Re: How to build parallel HDF5 with Intel MPI, Intel C++ and Intel Fortran

Matt Peterson
Ah, that's great news. Thank you for the script, your helpful comments, and your quick reply!

I have Intel XE 2013/2017 compilers. I've been interested in comparing the performance and ease-of-use of HDF5 to my current approach (using relational databases), and specifically for the Fortran-API. If I can get parallel HDF5 working properly in my program, it might eventually replace what I've got now. You've helped out a lot.

Thanks again,
Matt


On Wed, Mar 22, 2017 at 1:02 PM, Tony Garratt <[hidden email]> wrote:
​​Hi Matt

 Yes - I did manage to get it to work with Intel compilers on Windows - both serial and parallel builds. What compilers are you using?  Instructions for the Intel compilers are below.



Attached is a new CTestScript.cmake file.  Replace the one you are currently using with this one.

Also, you will need to edit the HDF5config.cmake file and add this
line to it. Change its value to match your Intel compiler:

set(CMAKE_GENERATOR_TOOLSET "Intel C++ Compiler 17.0")

(the name changes depending on what version of Intel compiler you are using - e.g. for XE2015 it is ""Intel C++ Compiler XE 15.0"). 

Also in the config file to build parallel add the following to the config file (this is on the HDF5 web page as well)

set(ADD_BUILD_OPTIONS "${ADD_BUILD_OPTIONS} -DHDF5_ENABLE_PARALLEL:BOOL=ON")
set(ADD_BUILD_OPTIONS "${ADD_BUILD_OPTIONS} -DMPIEXEC_MAX_NUMPROCS:STRING=4")

With Intel compilers I could not build the C++ library, so I added 

  set(ADD_BUILD_OPTIONS "${ADD_BUILD_OPTIONS} -DHDF5_BUILD_CPP_LIB:BOOL=OFF")

after the line

### uncomment/comment and change the following lines for other configuration options

in the config file.

I hope this helps. I have not built the parallel version with VS, only Intel, but I believe you should be OK with VS as long as you have MS MPI.

Regards,
Tony


On Wed, Mar 22, 2017 at 6:25 PM, Matt Peterson <[hidden email]> wrote:
Hi Tony,

Were you able to get this set up?

The HDF Cmake file continues to be a mystery to me and I've no time to figure it out right now. As an alternative, I've compiled the serial version on Windows using only the command line shell, without Visual Studio. However, if the parallel version compiled for you, I might give it another go.

Thanks,
MP

On Sun, Feb 26, 2017 at 2:11 PM, Tony Garratt <[hidden email]> wrote:

Thank you. This link also refers to using the -T option that I have used (with some partial success with Intel XE2015). I tried the suggestion on the page but it did not work for me. However, I think the way forward is to use -T option and then manually open the overall VS project and build manually.

Regards,
Tony

On Tue, Feb 21, 2017 at 7:35 PM, David Brooks <[hidden email]> wrote:
Hi Tony,

You might have already explored this approach but have you seen http://stackoverflow.com/questions/27623110/how-to-generate-a-visual-studio-project-that-uses-the-intel-compiler-using-cmake ??


Regards,
Dave

On 22/02/17 1:06 AM, Tony Garratt wrote:

Thank you for your reply. What I am wanting to do is build CGNS (https://cgns.github.io/WhatIsCGNS.html) in parallel to get parallel I/O, which requires HDF5 to be built in parallel - its not just thread safety I am after. 

Everything works fine on Linux, since there I can still use configure to build HDF5 and even Intel give advice on how to build it with their compilers at https://software.intel.com/en-us/articles/performance-tools-for-software-developers-building-hdf5-with-intel-compilers.

The problem is that the move to cmake has left me in the dark now. If configure was still supported on Windows for HDF5, then I the path forward is known. cmake appears to be very VS centric, which is fine if you are OK with MS compilers and not clear to me how I can switch to Intel compilers.

Regards,
Tony



On Tue, Feb 21, 2017 at 11:58 AM, Samer Afach <[hidden email]> wrote:
Hi Tony:

I don't know what the features of hdf5 with parallel are, but if all you want to achieve is have thread safety, I recommend that you use mutexes manually. Just wrap every call to hdf5 with std::lock_guard() or std::unique_lock(). If you can't use C++11, then you can get the same features from the boost library. This will make your code work independent of the build settings of hdf5.

I do this to avoid the obvious mistake of running a program in parallel with a version of hdf5 that is built for serial. It's much cleaner.

Cheers,
Sam

On February 21, 2017 12:23:03 PM GMT+01:00, Tony Garratt <[hidden email]> wrote:
Following on my previous post, I notice that at


you have built with Intel Fortran. How would one use cmake to build parallel HDF5 on Windows with both Intel C++ and Intel Fortran please?

Regards,
Tony

--
Dr Tony Garratt
Tel: <a href="tel:+44%2076%202430%209933" value="+447624309933" target="_blank">+44 7624 309933

--
Sent from my Android device.

_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5



--
Dr Tony Garratt
Tel: +44 7624 309933


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5



--
Dr Tony Garratt
Tel: <a href="tel:+44%2076%202430%209933" value="+447624309933" target="_blank">+44 7624 309933

_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5



--
Dr Tony Garratt
Tel: <a href="tel:+44%2076%202430%209933" value="+447624309933" target="_blank">+44 7624 309933

_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Loading...