HGCal Electronics Mapping for alpaka

Code format HGCalMappingESSourceTester.cc

Removing unused includes and delarations after PR comments

Correction: missing :: for namespace

Fix code style indentation

record descriptions

Re-introducing headers.h file needed for serialization classes

remove data and change locations to cms-data folder

moving to single record as all conds produced have the same IOV

re-introducing 1 record per payload, addressing additional code comments

moving to printf, removing unused product, and message logger

code checks

remove unused iomanip

adding digi SoAs

update collection names

moving emulators to SimCalorimetry/HGCalSimAlgos, adapting paths

moving digi soa filler to alpaka and adding extra tools

some progress with soafiller

updating

fixing but in module es producer (assignment of capture block index instead of capture block)

add dense index map for HGCalElectronicsId

add getMaxDataSize & getMaxERxSize functions

fix getDenseIndex bug; add typcode map to ModuleIndexer (to retrieve module dense index offset); use regexp for type code

add getFedAndModuleIndex; rename getDenseIndex -> getIndexForModule(Data) for elecID

add calibrations Alpaka ESProducer

rename ES_HGCal*Parameter.cc -> HGCal*ParameterHostCollection.cc for clarity

int -> uint32_t; uncomment debugging print out

clean

add HGCal configuration ESProducers + record + tester

rename ES_HGCal*Parameter.cc -> HGCal*ParameterHostCollection.cc for clarity

rename HGCal*ParameterHostCollection.cc -> HGCal*ParameterDeviceCollection.cc for clarity

rename 'HGCal.*(Device|Host)Collection -> (Device|Host)' for readability; move calib SoAs to CondFormats/HGCalObjects/; move TestHGCalRecHitESProducers to test

rename 'HGCal.*(Device|Host)Collection -> HGCal.*(Device|Host)' for readability

add HGCalRecHit

add DIGI2RECO calibration algorithms (see https://gitlab.cern.ch/hgcal-dpg/hgcal-comm/-/merge_requests/3)[C; remove detId/elecId from SoAs (should be redundant since common indexer); add HGCROCChannelDataFrame<HGCalDetId> to classes_def.xml

Move HGCalRawDataDefinitions.h.

[WIP] Refactor unpacker.

parse arrays of gain-dependent calibration parameters in eRx-blocks

add missing Noise attribute

updating usage of Rcd

fixing cfg files

Convert ECON-D payload to 32b words.

[Unpacker] rebase the unpacker on denseIdx and SoA digis

Update unpacker code.

SoA format of HGCal ECON-D header

Unpack header information to HGCalECONDInfoSoA; Add in quality checks for ECON-D

remove unused

add typecode regexp; pass typecode to modIndexer_.processNewModule

update ESProducers

update record

fix ESInputTags

prefix filename with T_EventSetup_

add HGCalConfiguration & HGCalConfigurationESProducer

Update and rename HGCalRecHitProducer.cc to HGCalRecHitProducers.cc

fill HGCal config structs for unpacker (header marks, charMode)

update calibration kernels

read HGCal FED & ECON-D configuration from JSON

Update unpacker to use HGCalConfiguration structure

make manual overrides truly optional

fix compilation issue due to initialization of const gain

Change to LogDebug

adding dense index info producer

adding hgcalmapping customise

fix misunderstanding in max data size expected from indexer

Implement TH2Poly in DQM based on 14e856f

Update HGCalConfigurationESProducer.cc

update unpacker and HGCalConfiguration

Add definitions of flags in digi SOA

update the other part to most recent commit

Add flags for passthrough ECON-Ds

Debug: loop through every denseIdx

Fix the number of channel in eRx;Add default flag for digi, representing digi not read in raw data

fully read config parameters from JSONs for config SoA; add sanity checks; add HGCalMappingModuleIndexer::getNFED; rename getFedAndModuleIndex -> getIndexForFedAndModule; clean code; ...

rename macro

silence debugging message in ESProducers

silence cout

add HGCalMappingModuleIndexer::getMaxModuleSize(fedid)

typo; SystemTestEventFilters/data -> Configuration/data

bugfix in the cm Sum

fixing dense index issue

Update comment in HGCalMappingParameterSoA.h

Some additional fixes (cms-sw#132)

- remove_if was missing a call to std::vector::erase to effectively remove unused entries
- typecodemap was missing a reset with the final indices
- usage of command line options was missing in module map tester
- do not use moduleLUT in DenseIndexInfoESProducer!! it has the full dimensions before trimming to the actual number of ECON-Ds in a FED

Add parsing function for econd flag; renaming HGCalECONDInfo to HGCalECONDPacketInfo

Add irot to module mapper (cms-sw#134)

* - remove_if was missing a call to std::vector::erase to effectively remove unused entries
- typecodemap was missing a reset with the final indices
- usage of command line options was missing in module map tester

* do not use moduleLUT in DenseIndexInfoESProducer!!

* Adding optional irot to module mapper file
- If existing it will be put in the module info SoA
- othwerwise a default irot=0 (baseline is used)

Suppress "unused variable" warning: remove CellIndexer in HGCalDigiSoaFiller.cc

apply code formats (cms-sw#135)

Apply suggestions from code review

Apply easy suggestions by Matti. (Looking into other comment & suggestions.)

Co-authored-by: Matti Kortelainen <[email protected]>

'const json' to fix compilation issue for Matti's suggestion

gain index should be taken as is, not subtracted by 1

make use of valid at rechit level / set rechit valids

fill rec hitflags

update kernel after closure

Implement suggestions for CMSSW_14_1_X PR cms-sw#45665 (cms-sw#137)

* Implement TH2Poly in DQM Services for HGCal DQM
* Apply changes from code quality checks
* Add a patch based on makortel's comments
* enum -> namespace + constexpr uint{8,16,32}_t
* Matti suggestion: remove '_t' postfix from HGCalFEDReadoutSequence_t, HGCalROCConfig_t, HGCalECONDConfig_t, HGCalFedConfig_t structs
* Matti suggestions & code checks: remove redundant comments; silence debugging cout or use MessageLogger; move includes; remove trailing '_' from function arguments
* Matti suggestion: reuse typecodeMap_.find() to avoid searching twice (TODO: throw exception if not found?)
* Matti suggestion: use SET_PORTABLEHOSTCOLLECTION_READ_RULES; add ClassVersion+checksum
* Matti suggestion: iRecord.getRecord<...>. -> iRecord.
* Matti suggestion: iRecord.getRecord<...>. -> iRecord.; rm redundant includes & couts; pass reference to fill_SoA_column to avoid copy
* Matti suggestions: use SET_PORTABLEHOSTCOLLECTION_READ_RULES for HGCalRecHitHost; Remove deprecated HGCalFlaggedECONDInfo* & comments;
* Matti suggestion: throw exception if type code not found; replace for-loop with std::accumulate
* Matti suggestion: remove TestObjects
* Matti suggestions: remove config as member data and produce unique_ptr instead of shared_ptr; NULL → nullptr; initialize optional override; memcp -> std::copy
* Matti suggestions: use file in path
* removing soafiller and updating buildfile (cms-sw#139)
* address comments on HGCalRecHitCalibrationAlgorithms.dev.cc
* irot as uint8_t instead of char
* restore DQM without TH2Poly
* apply code formats

---------

Co-authored-by: ywkao <[email protected]>
Co-authored-by: Pedro <[email protected]>

restore release base DQM classes (cms-sw#140)

move to chrono calls instead of struct inside namespace (cms-sw#141)

address comments (cms-sw#142)

update include (cms-sw#143)

remove ECON-D tests as they'll need to be re-written in a different form (cms-sw#144)

Declare HGCalRecHitCalibrationAlgorithms::calibrate as const method (cms-sw#145)

* remove ECON-D tests as they'll need to be re-written in a different form

* enable const

patching unpacker for padding of capture blocks within FED

move check of padding to start of capture block

Fix the throw, so the processing won't stop at the first event with error

Update HGCalRecHitCalibrationAlgorithms.dev.cc

CM sum -> 0.5*CM sum

Change the throws to specific returns

Optimize the producer

Debugging the padding of eRx subpackets

roll back (cms-sw#149)

apply code checks and remove dqm

CondFormats/DataRecord/interface/HGCalDenseIndexInfoRcd.h

@@ -0,0 +1,29 @@
+#ifndef CondFormats_HGCalDenseIndexInfoRcd_h
+#define CondFormats_HGCalDenseIndexInfoRcd_h
+// -*- C++ -*-
+//
+// Package:     CondFormats/DataRecord
+// Class  :     HGCalDenseIndexInfoRcd
+//
+/**\class HGCalDenseIndexInfoRcd HGCalDenseIndexInfoRcd.h CondFormats/DataRecord/interface/HGCalDenseIndexInfoRcd.h
+ *
+ * Description:
+ *   This record is used join information from the geometry and logical mapping
+ *   This record depends on the HGCalElectronicsMappingRcd and CaloGeometryRecord
+ *
+ */
+//
+// Author:      Pedro Da Silva, Izaak Neutelings
+// Created:     Mon, 29 May 2023 09:13:07 GMT
+//
+
+#include "FWCore/Framework/interface/DependentRecordImplementation.h"
+#include "FWCore/Utilities/interface/mplVector.h"
+#include "CondFormats/DataRecord/interface/HGCalElectronicsMappingRcd.h"
+#include "Geometry/Records/interface/CaloGeometryRecord.h"
+
+class HGCalDenseIndexInfoRcd : public edm::eventsetup::DependentRecordImplementation<
+                                   HGCalDenseIndexInfoRcd,
+                                   edm::mpl::Vector<HGCalElectronicsMappingRcd, CaloGeometryRecord> > {};
+
+#endif

CondFormats/DataRecord/interface/HGCalModuleConfigurationRcd.h

@@ -0,0 +1,28 @@
+#ifndef CondFormats_HGCalModuleConfigurationRcd_h
+#define CondFormats_HGCalModuleConfigurationRcd_h
+// -*- C++ -*-
+//
+// Package:     CondFormats/DataRecord
+// Class  :     HGCalModuleConfigurationRcd
+//
+/**\class HGCalModuleConfigurationRcd HGCalModuleConfigurationRcd.h CondFormats/DataRecord/interface/HGCalModuleConfigurationRcd.h
+ *
+ * Description:
+ *   This record is used for passing the configuration parameters to the calibration step in RAW -> RECO,
+ *   This record depends on the HGCalMappingModuleIndexerRcd.
+ *
+ */
+//
+// Author:      Pedro Da Silva, Izaak Neutelings
+// Created:     Mon, 29 May 2023 09:13:07 GMT
+//
+
+#include "FWCore/Framework/interface/DependentRecordImplementation.h"
+#include "FWCore/Utilities/interface/mplVector.h"
+#include "CondFormats/DataRecord/interface/HGCalElectronicsMappingRcd.h"
+
+class HGCalModuleConfigurationRcd
+    : public edm::eventsetup::DependentRecordImplementation<HGCalModuleConfigurationRcd,
+                                                            edm::mpl::Vector<HGCalElectronicsMappingRcd> > {};
+
+#endif

CondFormats/DataRecord/src/HGCalDenseIndexInfoRcd.cc

@@ -0,0 +1,15 @@
+// -*- C++ -*-
+//
+// Package:     CondFormats/DataRecord
+// Class  :     HGCalDenseIndexInfoRcd
+//
+// Implementation:
+//     [Notes on implementation]
+//
+// Author:      Pedro Da Silva, Izaak Neutelings
+// Created:     Mon, 29 May 2023 09:13:07 GMT
+
+#include "CondFormats/DataRecord/interface/HGCalDenseIndexInfoRcd.h"
+#include "FWCore/Framework/interface/eventsetuprecord_registration_macro.h"
+
+EVENTSETUP_RECORD_REG(HGCalDenseIndexInfoRcd);

CondFormats/DataRecord/src/HGCalModuleConfigurationRcd.cc

@@ -0,0 +1,15 @@
+// -*- C++ -*-
+//
+// Package:     CondFormats/DataRecord
+// Class  :     HGCalModuleConfigurationRcd
+//
+// Implementation:
+//     [Notes on implementation]
+//
+// Author:      Pedro Da Silva, Izaak Neutelings
+// Created:     Mon, 29 May 2023 09:13:07 GMT
+
+#include "CondFormats/DataRecord/interface/HGCalModuleConfigurationRcd.h"
+#include "FWCore/Framework/interface/eventsetuprecord_registration_macro.h"
+
+EVENTSETUP_RECORD_REG(HGCalModuleConfigurationRcd);

CondFormats/HGCalObjects/BuildFile.xml

@@ -11,4 +11,4 @@
 <flags ALPAKA_BACKENDS="1"/>
 <export>
   <lib name="1"/>
-</export>
+</export>

CondFormats/HGCalObjects/interface/HGCalCalibrationParameterHost.h

@@ -0,0 +1,19 @@
+#ifndef CondFormats_HGCalObjects_interface_HGCalCalibrationParameterHost_h
+#define CondFormats_HGCalObjects_interface_HGCalCalibrationParameterHost_h
+
+#include "DataFormats/Portable/interface/PortableHostCollection.h"
+#include "CondFormats/HGCalObjects/interface/HGCalCalibrationParameterSoA.h"
+
+namespace hgcalrechit {
+
+  // SoA with channel-level calibration parameters in host memory:
+  //   pedestal, CM_slope, CM_ped, BXm1_kappa
+  using HGCalCalibParamHost = PortableHostCollection<HGCalCalibParamSoA>;
+
+  // SoA with ROC-level configuration parameters in host memory:
+  //   gain
+  using HGCalConfigParamHost = PortableHostCollection<HGCalConfigParamSoA>;
+
+}  // namespace hgcalrechit
+
+#endif  // CondFormats_HGCalObjects_interface_HGCalCalibrationParameterHost_h

CondFormats/HGCalObjects/interface/HGCalCalibrationParameterSoA.h

@@ -0,0 +1,47 @@
+#ifndef CondFormats_HGCalObjects_interface_HGCalCalibrationParameterSoA_h
+#define CondFormats_HGCalObjects_interface_HGCalCalibrationParameterSoA_h
+
+#include <Eigen/Core>
+#include <Eigen/Dense>
+
+#include "DataFormats/SoATemplate/interface/SoACommon.h"
+#include "DataFormats/SoATemplate/interface/SoALayout.h"
+#include "DataFormats/SoATemplate/interface/SoAView.h"
+
+//#include "CondFormats/HGCalObjects/interface/HGCalMappingModuleIndexer.h"
+
+namespace hgcalrechit {
+
+  // human-readable data type to replace bool for memcpy of vector<>::.data()
+  // NOTE: bool and byte have the same memory size
+  using mybool = std::byte;
+
+  // Generate structure of channel-level arrays (SoA) layout with RecHit dataformat
+  GENERATE_SOA_LAYOUT(HGCalCalibParamSoALayout,
+                      SOA_COLUMN(float, ADC_ped),     // ADC pedestals, O(91)
+                      SOA_COLUMN(float, Noise),       // noise, O(3)
+                      SOA_COLUMN(float, CM_slope),    // common mode slope, O(0.25)
+                      SOA_COLUMN(float, CM_ped),      // common mode pedestal (offset), O(92)
+                      SOA_COLUMN(float, BXm1_slope),  // leakage correction from previous bunch, O(0.0)
+                      SOA_COLUMN(float, ADCtofC),  // ADC conversion to charge (fC), depends on gain (80, 160, 320 fC)
+                      SOA_COLUMN(float, TOTtofC),  // TOT conversion to charge (fC), depends on gain (80, 160, 320 fC)
+                      SOA_COLUMN(float, TOT_ped),  // TOT pedestal (offset), O(9.0)
+                      SOA_COLUMN(float, TOT_lin),  // threshold at which TOT is linear, O(200)
+                      SOA_COLUMN(float, TOT_P0),   // coefficient pol2 in nonlinear region, O(145)
+                      SOA_COLUMN(float, TOT_P1),   // coefficient pol2 in nonlinear region, O(1.0)
+                      SOA_COLUMN(float, TOT_P2),   // coefficient pol2 in nonlinear region, O(0.004)
+                      SOA_COLUMN(float, TOAtops),  // TOA conversion to time (ps)
+                      SOA_COLUMN(float, MIPS_scale),  // MIPS scale
+                      SOA_COLUMN(mybool, valid)       // if false: mask dead channel
+  )
+  using HGCalCalibParamSoA = HGCalCalibParamSoALayout<>;
+
+  // Generate structure of ROC-level arrays (SoA) layout with RecHit dataformat
+  GENERATE_SOA_LAYOUT(HGCalConfigParamSoALayout,
+                      SOA_COLUMN(uint8_t, gain)  // for ADC to charge (fC) conversion (80, 160, 320 fC)
+  )
+  using HGCalConfigParamSoA = HGCalConfigParamSoALayout<>;
+
+}  // namespace hgcalrechit
+
+#endif  // CondFormats_HGCalObjects_interface_HGCalCalibrationParameterSoA_h

CondFormats/HGCalObjects/interface/HGCalConfiguration.h

@@ -0,0 +1,68 @@
+// Authors: Izaak Neutelings (May 2024)
+// Sources: https://docs.google.com/spreadsheets/d/13G7sOjssqw4B5AtOcQV3g0W01oZUOMM6Hm_DduxBEPU
+#ifndef CondFormats_HGCalObjects_HGCalConfiguraton_h
+#define CondFormats_HGCalObjects_HGCalConfiguraton_h
+#include "CondFormats/Serialization/interface/Serializable.h"
+#include "CondFormats/HGCalObjects/interface/HGCalMappingModuleIndexer.h"
+#include <map>
+#include <vector>
+
+// @short configuration for ECON eRX (one half of HGROC)
+struct HGCalROCConfig {
+  uint32_t charMode;  // characterization mode; determines data fields in ROC dataframe
+  uint8_t gain;       // pre-amp gain used (1: 80 fC, 2: 160 fC, 4: 320 fC)
+  //uint32_t clockPhase;     // fine adjustment of the phase within the 40 MHz
+  //uint32_t L1AcceptOffset; // coarse adjustment to get the peak in the right place
+  //uint32_t injChannels;    // injected channels for injection scan: 2b word to identify if connected or not+info no capacitor chosen
+  //uint32_t injCharge;      // injected charge for injection scan: convert it to a float in units of fC offline (DAC setting?)
+  COND_SERIALIZABLE;
+};
+
+// @short configuration for ECON-D module
+struct HGCalECONDConfig {
+  //std::string typecode;
+  uint32_t headerMarker;  // begin of event marker/identifier for ECON-D
+  uint32_t passThrough;   //pass through mode (this is just as check as it'll be in the ECON-D header anyway)
+  std::vector<HGCalROCConfig> rocs;
+  COND_SERIALIZABLE;
+};
+
+// @short configuration for FED
+struct HGCalFedConfig {
+  bool mismatchPassthroughMode;  // ignore ECON-D packet mismatches
+  uint32_t cbHeaderMarker;       // begin of event marker/identifier for capture block
+  uint32_t slinkHeaderMarker;    // begin of event marker/identifier for S-link
+  //uint32_t delay; // delay
+  std::vector<HGCalECONDConfig> econds;
+  COND_SERIALIZABLE;
+};
+
+/**
+ *  @short Main HGCal configuration with a tree structure of vectors of
+ *         HGCalFedConfig/HGCalECONDConfig/HGCalROCConfig structs as follows:
+ %         config.feds[dense_fed_idx].econds[dense_econd_idx].rocs[dense_eRx_idx]
+ **/
+class HGCalConfiguration {
+public:
+  std::vector<HGCalFedConfig> feds;
+  //friend std::ostream& operator<< (std::ostream&, const HGCalConfiguration&);
+
+private:
+  COND_SERIALIZABLE;
+};
+
+inline std::ostream& operator<<(std::ostream& os, const HGCalConfiguration& config) {
+  uint32_t nfed = config.feds.size();
+  uint32_t ntotmod = 0;
+  uint32_t ntotroc = 0;
+  for (auto const& fed : config.feds) {
+    ntotmod += fed.econds.size();  // number of ECON-D modules for this FED
+    for (auto const& mod : fed.econds) {
+      ntotroc += mod.rocs.size();  // number of eRx half-ROCs for this ECON-D module
+    }
+  }
+  os << "HGCalConfiguration(nfed=" << nfed << ",ntotmod=" << ntotmod << ",ntotroc=" << ntotroc << ")";
+  return os;
+}
+
+#endif